Senior Databricks Data Engineer - Data Platform & Medallion Architecture 3657895

Job Description

Job Description

Be Part Of A High-Performing Team

Join a data engineering organization that is modernizing enterprise data capabilities and building scalable solutions to improve how critical business information is ingested, standardized, governed, and consumed. The team is expanding its use of Databricks to create reusable engineering frameworks, modern data models, observability capabilities, and advanced analytics solutions.

This role will work closely with architects, technical leads, business partners, and distributed engineering teams while contributing directly to new Databricks capabilities. Projects include data pipeline modernization, medallion architecture, underwriting-data analytics, data-quality frameworks, and emerging AI-enabled capabilities such as intelligent document processing.

What's In Store For You

• Engagement: W2 only (no C2C/1099)
• Work model: Fully remote within the U.S., working Eastern Time hours
• Preference for candidates located in or near Richmond, Virginia
• Contract-to-hire opportunity
• Opportunity to help define reusable Databricks engineering standards and frameworks used across multiple teams
• Exposure to newer Databricks capabilities, including AI-assisted development and intelligent document processing
• Broad ownership across requirements, development, testing, deployment, documentation, and production support

How You Will Make An Impact

• Design, develop, and enhance enterprise-scale data solutions using Databricks and medallion architecture.
• Build reusable frameworks and common functions that standardize data ingestion, transformation, validation, repair, and pipeline monitoring.
• Develop Bronze, Silver, and Gold data layers that improve the usability, consistency, and accessibility of enterprise data.
• Create dimensional data structures, including facts, dimensions, derived attributes, and SCD Type 2 implementations.
• Develop and orchestrate both batch and streaming data pipelines.
• Partner with business stakeholders to translate high-level data requirements into technical solutions.
• Collaborate with architects and technical leadership on solution design and engineering standards.
• Perform hands-on development, unit testing, system integration testing, debugging, and production support.
• Build solutions that improve pipeline observability, SLA monitoring, data quality, and operational transparency.
• Optimize Databricks compute and storage performance.
• Develop documentation, training materials, and reusable guidance so other engineering teams can adopt frameworks and capabilities.
• Explore newer Databricks functionality such as Intelligent Document Processing, Genie, and self-service workspace capabilities and establish practical implementation patterns.

Do You Have the Expertise to Lead in Databricks Data Engineering?

• Minimum 3 years of full-time, hands-on Databricks experience within Azure or AWS environments.
• Strong experience implementing the medallion architecture , including Bronze, Silver, and Gold data layers.
• Expert-level SQL skills.
• Intermediate or stronger PySpark programming experience.
• Demonstrated experience creating reusable data engineering frameworks, common functions, or standardized pipeline components.
• Hands-on experience creating dimensional models, fact tables, dimensions, derived business values, and SCD Type 2 structures.
• Strong Databricks orchestration experience supporting batch and streaming pipelines.
• Experience with Databricks capabilities such as Unity Catalog, Spark UI, job scheduling, Lakeflow, and data quality expectations .
• Proven experience with unit testing, SIT, UAT support, debugging, and troubleshooting data defects.
• Experience optimizing Databricks compute and storage.
• Strong understanding of CI/CD, Git-based development, and automated deployment processes .
• Experience working within Agile environments, including user stories, technical task definition, and effort estimation.
• Ability to communicate effectively with business users, architects, engineering teams, and onshore/offshore partners.
• Strong problem-solving, design-thinking, teamwork, and documentation skills.
• Candidates with 5+ years of Databricks experience and experience building major platform capabilities from scratch are strongly preferred.
• Insurance data, data governance, or data observability experience is beneficial.