Sr. Databricks Migration Engineer

Senior Databricks Migration Engineer – SQL Server/ADF Dexian Government Solutions Location: 100% Remote (United States) Salary: $145,000–$165,000 annually Contract: Funded U.S. Postal Service (USPS) contract Citizenship and Suitability: U.S. citizenship is required by the federal contract. Candidates must meet the applicable USPS background investigation and suitability requirements before beginning work. Position Overview Dexian Government Solutions is seeking an experienced Senior Databricks Migration Engineer to lead an enterprise data modernization initiative supporting the U.S. Postal Service (USPS). This position offers the opportunity to transform legacy SQL Server and Azure Data Factory (ADF) workloads into a scalable, secure, high-performance Databricks Lakehouse environment built on Delta Lake. The successful candidate will serve as the primary technical authority for migration architecture, data pipeline engineering, performance and cost optimization, security and governance, and knowledge transfer to the USPS team. This position requires demonstrated migration experience. We are seeking a hands-on technical leader who has successfully migrated production SQL Server stored procedures and ADF pipelines into Databricks. Experience limited to developing new (greenfield) Databricks pipelines will not satisfy the core migration requirement. Key Responsibilities Migration and Lakehouse Architecture Lead the migration of legacy SQL Server stored procedures (T-SQL) and Azure Data Factory (ADF) pipelines to the Databricks Lakehouse, refactoring procedural logic into PySpark and Spark SQL. Plan and execute migration validation activities, including parallel runs, data reconciliation, and controlled cutover from legacy to Lakehouse workloads. Design and implement Bronze, Silver, and Gold data layers following Medallion Architecture principles. Translate traditional relational data warehouse designs into scalable, distributed, cloud-native Lakehouse architectures. Develop reusable, production-ready ETL/ELT frameworks using PySpark, Lakeflow Declarative Pipelines (formerly Delta Live Tables/DLT), and Lakeflow Jobs (formerly Databricks Workflows). Develop metadata-driven ingestion and data quality frameworks supporting incremental loading at enterprise scale. Design and optimize dimensional models and star schemas in the Gold layer to support enterprise analytics and reporting. Performance Engineering and Cost Optimization Optimize Databricks SQL Warehouses for high-concurrency, low-latency Power BI workloads using DirectQuery and Import modes. Apply advanced Delta Lake optimization techniques, including liquid clustering, Z-ordering where applicable, data skipping, and materialized views. Establish partitioning strategies, file-size management, and compaction standards to improve processing efficiency. Define compute governance standards covering cluster sizing, autoscaling, and serverless SQL compute. Develop monitoring dashboards to track Databricks Unit (DBU) consumption, workload performance, and cost-reduction opportunities. Security and Data Governance Design and implement centralized data governance using Databricks Unity Catalog. Implement row-level and column-level security controls to protect sensitive data and support compliant access. Integrate Lakehouse security architecture with Microsoft Entra ID (formerly Azure Active Directory) and Role-Based Access Control (RBAC). Provide analysts and Power BI consumers with secure, governed, and reliable access to enterprise data. Technical Leadership and Knowledge Transfer Serve as the primary technical lead for the migration initiative, providing architectural direction and hands-on engineering expertise. Conduct pair-programming sessions, technical workshops, and code reviews to strengthen internal Databricks and Spark capabilities. Mentor engineers transitioning from traditional SQL-centric development to distributed PySpark and Lakehouse engineering. Develop comprehensive technical documentation, including architecture diagrams, design patterns, technical standards, and optimization playbooks. Establish a structured knowledge transfer framework to ensure the USPS team can independently maintain and enhance the environment following migration. Communicate technical decisions, progress, risks, and recommendations effectively to technical and nontechnical stakeholders. Manage responsibilities in an organized, timely manner with strong attention to detail. Required Qualifications U.S. citizenship is mandatory for this position due to federal contract requirements. Candidates must meet the applicable USPS Public Trust background investigation and suitability requirements before the start date. Bachelors degree or higher from an accredited college or university in Computer Science, Engineering, or a related technical field. At least 5 years of experience in data engineering, data system development, or a related role. At least 5 years of experience with cloud platforms such as Microsoft Azure, AWS, or GCP. At least 1 year of experience leading complex, cross-functional data engineering projects or technical teams. Demonstrated hands-on experience migrating production SQL Server stored procedures and Azure Data Factory pipelines into Databricks. Strong hands-on experience with Databricks Lakehouse, Apache Spark, PySpark, Spark SQL, Delta Lake, Medallion Architecture, and cloud-native data architectures. Experience building production ETL/ELT pipelines on distributed platforms using Lakeflow Declarative Pipelines (formerly Delta Live Tables/DLT) and Lakeflow Jobs (formerly Databricks Workflows). Experience with data warehousing, dimensional modeling, star schemas, enterprise data lakes, incremental loading, and metadata-driven ingestion and data quality frameworks using PySpark. Experience implementing data governance and access controls using Unity Catalog, Microsoft Entra ID, and RBAC, including row-level and column-level security. Experience optimizing Delta Lake and Databricks SQL Warehouse performance, with knowledge of techniques such as liquid clustering, Z-ordering, data skipping, partitioning, and file compaction. Strong understanding of cloud-native databases, storage solutions, and distributed compute environments. Demonstrated technical leadership, including code reviews, mentoring engineers, technical documentation, and knowledge transfer. Excellent analytical, troubleshooting, organizational, documentation, and communication skills. Preferred Qualifications Experience optimizing Power BI DirectQuery and Import performance against Databricks SQL Warehouses. Experience with Databricks cluster management, autoscaling, serverless compute, and DBU cost monitoring and optimization. Experience implementing CI/CD for Databricks using Databricks Asset Bundles, Azure DevOps, GitHub Actions, or comparable tools. Databricks Certified Data Engineer Professional or Associate certification. Experience migrating other legacy data warehouse platforms to Databricks, including Teradata, Oracle, SSIS, or Informatica. Why Join Us? 100% Remote: Work remotely from the United States while supporting a major federal organization. Funded Contract: Join an established, funded USPS engagement. Competitive Compensation: $145,000–$165,000 annually. Technical Ownership: Lead the migration from legacy SQL Server and ADF workloads to a modern Databricks Lakehouse. Architectural Impact: Influence enterprise data architecture, performance, security, and governance. Leadership Opportunity: Develop the internal teams long-term Databricks engineering capabilities through mentorship and knowledge transfer. Meaningful Work: Modernize the data infrastructure supporting analytics and operational decision-making for one of the nations largest federal organizations. Company Description Dexian Government Solutions is an award-winning ISO 9001:2015 certified business and GSA contract holder providing diversified Information Technology services to both Civilian and Defense agencies. Services include Software Development, Systems Integration, Data Management, Project Management, Operations & Maintenance, Cybersecurity, and Training and Audio/Visual (AV) Solutions. Dexian Government Solutions has received several recognitions, including rankings on \"Top 50 Companies to Watch\", Washington Technologys Annual \"FAST 50\", and Inc. 500s List of \"Fastest Growing Private Companies\". The Dexian Government Solutions team is comprised of individuals who are dedicated to the success and sustainability of our customers and their missions. Our combination of technical expertise, big business experience, and small business agility allows us to promptly provide our customers with exceptional IT and engineering solutions. Benefits Our robust benefits package includes Open Paid Time Off, 11 Federal Paid Holidays & 5 Paid Sick Days, Company-paid Life/AD&D, Company-paid Short Term and Long-Term Disability, Health Insurance with Company Contribution, 401k Plan with Company Match, Employee Recognition Program, opportunity for Employee Referral Bonus, opportunity for annual Performance Bonus and much more! EEO Statement Dexian Government Solutions is proud to be an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All employment is decided based on qualifications, merit, and business need. All applicants will be considered for employment without attention to race, religion, color, national origin, ancestry, physical or mental disability, medical condition, pregnancy (including childbirth, lactation and related medical conditions), marital status, genetic information (including characteristics and testing), gender, sexual orientation, gender identity or expression, military and veteran status, or any other status protected under federal, state, or local law in the locations where we operate. If you are an individual with a disability and would like to request a reasonable accommodation as part of the employment selection process, please contact Human Resources. The Company invites any applicant and/or employee to review the Companys written Affirmative Action Plan. This plan is available for inspection upon request. Dexian