Senior Databricks Migration Engineer
PlanIT Group
Job Description
Job Description
Senior Databricks Migration Engineer
We are seeking a Senior Databricks Migration Engineer to join our team of qualified, diverse individuals supporting a Federal Government customer in Rosslyn, VA. Candidates MUST be US citizens. The individual serves as the authoritative resource for the agency who specializes in preparing big data infrastructure for analytical or operational uses. They are responsible for designing and creating systems that collect, manage, and convert raw data into usable information for data scientists and business analysts to interpret and enables the agency to make smarter decisions and optimize operations. Position RequirementsPrimary Requirements
• Lead the technical migration from legacy SQL Server stored procedures and ADF pipelines to Databricks Lakehouse (Delta Lake), ensuring best practice Lakehouse design.
• Translate traditional relational data warehousing paradigms into scalable, distributed Lakehouse frameworks (Bronze, Silver, Gold).
• Design robust, reusable ETL/ELT frameworks using PySpark, Delta Live Tables (DLT), and Databricks Workflows.
• Architect and refine the Gold Layer (dimensional models, star schemas) specifically to maximize Power BI performance.
• Optimize Databricks SQL Warehouses to support high-concurrency, low-latency Power BI queries (DirectQuery and Import modes).
• Implement advanced optimization techniques, including Z-Ordering, data skipping, liquid clustering, and materialized views.
• Define and enforce governance standards for cluster sizing, auto-scaling policies, and serverless SQL compute to balance performance with cost.
• Implement proactive monitoring dashboards to track Databricks Unit (DBU) consumption and identify cost-saving opportunities.
• Establish best practices for partition strategies and file size management within Delta Lake.
• Design and implement a robust data security model using Unity Catalog for centralized governance.
• Enforce row-level and column-level security policies to ensure compliant data access for Power BI consumers and internal analysts.
• Align the Lakehouse security architecture with existing enterprise Azure Active Directory (Microsoft Entra ID) and RBAC standards.
• Act as the primary technical lead, conducting dedicated pair-programming sessions, workshops, and code reviews to transition the team from SQL-centric to Spark-centric thinking.
• Create comprehensive technical documentation, including architecture diagrams, design patterns, and optimization playbooks.
• Build a foundational knowledge transfer framework to ensure the internal team is fully self-sufficient post-migration.
• Communicate effectively verbally and in written form to both technical and non-technical audience
• Work in an organized fashion, completing tasks timely while paying close attention to details Desired KSAs
• Mastery of data engineering principles, including data modeling, ETL (Extract, Transform, Load) processes, and data pipelines
• Proficiency with Azure Data Lake data storage and processing services
• Skilled at designing, building, and optimizing data pipelines for ingesting, transforming, and loading data
• Proficiency in languages such as SQL and Python/PySpark for data manipulation and pipeline development
• Skilled at identifying and resolving data-related challenges
• Skilled at creating efficient data models that meet business requirements.
• Skilled at optimizing query performance and system scalability Experience
• Must have a Bachelor's degree or higher from an accredited college or university in Computer Science, Engineering, or a related technical field
• 5 years' experience in data engineering, data system development or related roles
• 5 years' experience with cloud platforms (e.g. Azure, AWS, GCP)
• 1 year leading complex, cross-functional data projects and technical teams
• Experience with Databricks Lakehouse, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms
• Experience with data warehousing, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark
• Candidates MUST be a US Citizen and be able to obtain a Position of Public Trust Clearance. Must not have traveled outside the US for a combined total of 6 months or more in last 5 years and have resided in the US for the last 5 years
Vacancy posted 21 days ago
Similar jobs that could be interesting for youBased on the Senior Databricks Migration Engineer in Reston, VA vacancy
$123k - $161k
...call home.Job DescriptionTHE IMPACT YOU WILL MAKEThe Senior Associate - Palantir Forward Deployed Engineer(FDE), partners with business stakeholders,... ...operational solutions.Exposure to Palantir Foundry, Databricks, Snowflake, AWS, or similar modern data platforms.Desired...DatabricksSeniorFull timeWork at officeRemote work- ...and technology. To support this mission, we are seeking a Senior Cybersecurity Engineer to protect and defend critical enterprise and mission IT... ...certification). Desired Experience Familiarity with DataBricks, GitLab, or Jira for managing security workflows....DatabricksSeniorContract work
- ...technology. To support this mission, we are seeking a Senior Information System Security Engineer (ISSE) to provide expert technical and programmatic... ...DevSecOps pipelines. Operational experience with DataBricks, GitLab, or Jira in a secure environment. Experience...DatabricksSeniorContract work
- ...technology. To support this mission, we are seeking a QA Senior Software Engineer to serve as a recognized authority within the company. In... ...Familiarity with testing data-heavy applications involving Spark or DataBricks. Previous experience working within a government...DatabricksSeniorContract work
- ...seeking an Identity, Credential, and Access Management (ICAM) Engineer to develop, integrate, and maintain enterprise identity services... ...identity attributes and access entitlements. • Support migration from legacy identity platforms to modern services, preserving...SeniorFull timeTemporary work
- ...Lead the end-to-end data workstream and establish the migration strategy, governance cadence, priorities, and... ...activities. ~ Oversee migration execution across Databricks architecture, engineering, testing, release, stabilization, and operational transition...Databricks
- ...a skilled Delivery Consultant to join the Cloud Platform and Migrations team. In this role, you'll work closely with customers to design... ...system experience- Bachelor's degree in computer science, engineering, mathematics or equivalent- 3+ years of customer-facing work,...Senior
$81.5k - $138.55k
...continues to grow, the team is focused on migrating legacy workloads, streamlining... ...solutions at scale. The test automation engineer implements automated testing, release validation... ...platforms and services including Azure, Databricks, APIs, workflow orchestration platforms,...DatabricksFull timeContract workWork experience placementLive inWork at officeImmediate startRemote work- ...Job Description Job Description Job Description: Senior Principal Cyber Systems Engineer (DevOps Engineer) The selected candidate will integrate... ...debug critical issues. Candidate will also support the migration of containerized workflows from platforms such as Docker...Senior
- ...Position Overview: We are seeking an experienced Senior Cloud Automation Engineer to support a large-scale enterprise cloud transformation initiative... ...on-premises VMware infrastructure and is looking to migrate substantial workloads to the cloud while developing new cloud...Senior
$109.05 per hour
...Job Description Job Description Senior AWS Cloud Platform Engineer Location: Reston, VA (Hybrid) Duration: 12-Month Contract Pay Rate:... ...role focused on AWS infrastructure engineering, cloud migration, automation, platform support, troubleshooting, and production...SeniorContract work$123k - $161k
...applications under limited supervision.THE IMPACT YOU WILL MAKEThe Senior Windows Endpoint Engineer role will offer you the flexibility to make each day... ...policies, update rings, and security baselines. Support migration and coexistence strategies between Workspace ONE and...SeniorFull timeWork at officeRemote work- ...Contract Job #3999 Title: Software Engineer Location: Herndon, Virginia... ...with mission operators, analysts, and senior stakeholders to define requirements and... ...data technologies (e.g., Spark, Hadoop, Databricks). Experience with transformer-based models...DatabricksContract work
- ...Senior Principal ArchitectIf you are excited about shaping the... ...coaching to domain architects and engineering leads; intervenes directly... ...technologies (Kafka, Databricks, Spark, Snowflake or equivalent... ...modernization, including legacy migration and strangler-fig...DatabricksSenior
- ...Job Description Job Description Job Description acuCyber is looking for a Senior Computer Engineer to lead the design, development, integration, and evaluation of complex computer systems and networks. The Senior Computer Engineer will translate operational requirements...SeniorTemporary work
$154.9k - $248.2k
...platforms. The Opportunity: We are seeking a seasoned Fullstack Senior Staff Software Engineer with experience building GenAI-enabled applications. We'... ...or more years of experience with a PhD• Experience with Databricks/Spark pipelines, MLFlow, or modern MLOps tools.•...DatabricksSeniorFull timeWork experience placementWork at officeLocal area- ...solutions connecting the space, air, land, sea and cyber domains in the interest of national security. Job Title: Senior Specialist, System Engineering Job Code: 44598 Job Location: Herndon, Virginia Job Schedule: 9/80 (Every Other Friday Off!) Job...SeniorFull time
- ...Job DescriptionJob DescriptionSenior Java Full Stack Engineer (AWS Cloud Migration)Location: Reston, VA (Hybrid) Type: Contract through December... ...approximately once per week or biweekly.OverviewWe are seeking a Senior Java Full Stack Engineer to support a high-profile...SeniorContract workLocal areaImmediate start1 day per week
$164.3k - $222.3k
...ability to grow your career.We are seeking an experienced Senior Storage Engineer to implement, operate, and continuously improve highly available... ...shell scriptingExperience performing large-scale storage migrations and technology refreshesExperience working in environments...SeniorWork at officeFlexible hours$74.09k - $125.95k
...grow, the team is focused on migrating legacy workloads,... ...at scale. We are seeking a Senior FinOps Analyst to provide specialized... ...closely with Product Operations, engineering, data, and technical leads... .... Analyze Azure and Databricks consumption and cost patterns...DatabricksSeniorFull timeContract workWork experience placementLive inH1bWork at officeImmediate startRemote work$99k - $225k
...Intelligence Solutions Architect, Senior The Opportunity:... ...Bedrock, PySpark, Synapse, or Databricks ~1+ years of experience designing... ...architecture, platform engineering, and development teams on... ...of Big Data solutions, data migration technologies, and processes...DatabricksSeniorFull timeContract workPart timeLocal areaRemote work- ...Overview We are seeking a highly skilled Senior ID & Access Management Engineer to serve as a Technical Lead and Product Owner supporting... ...Experience leading large-scale IAM modernization or migration initiatives. Experience with Kubernetes, Docker, Azure...SeniorInternshipMonday to Friday
$160k - $220k
...future is our shared goal. AI/ML Engineer Cyber Threat Hunting Required Skills... ...engineering technologies such as Spark Databricks SQL graph databases or distributed analytics... .... Please apply for immediate consideration. Required Experience: Senior IC...DatabricksSeniorFull timeTemporary workLocal areaImmediate start$160k - $200k
...: SAIC seeks a SeniorCloud Computing Engineer to join our team of professionals providing... ..., at sea, in air, and in space. The Senior Cloud Computing Engineer sought here... ...architecture, and application and data migration Active member of enterprise R&D, learning...SeniorFull timeWork experience placementWork at officeRemote workShift work$99k - $225k
...Intelligence Solutions Architect, Senior The Opportunity:... ...Bedrock, PySpark, Synapse, or Databricks ~5+ years of experience designing... ...architecture, platform engineering, and development teams on... ...of Big Data solutions, data migration technologies, and processes...DatabricksSeniorFull timeContract workPart timeWork at officeLocal areaRemote work$106.5k - $197.5k
...solutions connecting the space, air, land, sea and cyber domains in the interest of national security. Job Title: Senior Specialist, Electrical Engineer Job Code: 47188 Job Location: Palm Bay, FL Job Schedule: 9/80 - Employees work 9 out of every 14 days - totaling 80...SeniorLocal areaImmediate startFlexible hours$81.5k - $138.55k
...continues to grow, the team is focused on migrating legacy workloads, streamlining... ...solutions at scale. The Data Integration Engineer implements data movement, replication, APIs... ...frameworks, and migration solutions utilizing Databricks, Delta Lake, Azure Data Factory, Spark,...DatabricksFull timeContract workWork experience placementLive inH1bWork at officeImmediate startRemote work$150k - $185k
...individual to join our team as a Senior Data Architect to own the... ...data challenges, and mentoring engineers and junior architects. The... ...design for a cloud data platform migration, defining the target-state... ...warehousing and lake technologies (Databricks, Redshift, BigQuery, or...DatabricksSeniorContract workWork at officeLocal area3 days per week$91k - $136k
...Job Description Job Description ATCS is seeking a Senior Roadway Engineer to join our growing Virginia transportation team. This is a hybrid position based in our Herndon headquarters offering the opportunity to work on high-profile VDOT and municipal roadway projects...SeniorFull timeFor contractorsLocal areaImmediate startFlexible hours$103.8k - $218.1k
Job Title: Test Engineer (Senior) Job Category: Engineering Time Type: Full time Minimum Clearance Required to Start: Secret Employee... ...systems Perform comprehensive testing for cloud migration efforts ensuring FedRAMP compliance and data sovereignty requirements...SeniorFull timeContract workWork experience placementLocal areaFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior Databricks Migration Engineer. Be the first to apply!
Related searches
- senior living Reston, VA
- senior accountant manufacturing Reston, VA
- senior manager customer operations Reston, VA
- senior full stack web developer Reston, VA
- senior magento developer Reston, VA
- senior sas administrator Reston, VA
- senior security analyst Reston, VA
- senior application support engineer Reston, VA
- senior software design engineer Reston, VA
- senior devops cloud engineer Reston, VA





