Azure Databricks Data Engineer

Job Title: Azure Databricks Data Engineer
Location: Saint Louis, MO
Hire Type: Full time

Candidates should have strong hands-on experience with:

• Significant Data Engineering experience with substantial Azure Databricks development.
• Azure Databricks production-grade batch pipelines and Lakehouse solutions.
• PySpark / Spark SQL advanced transformations, joins, window functions, reusable frameworks, debugging and performance tuning.
• Delta Lake table design, schema evolution, merge/upsert, optimization, data reliability and history management.
• Medallion Architecture Bronze/source-aligned, Silver/conformed and Gold/consumption-ready layers.
• Data Modeling fact/dimension design, conformed dimensions, SCDs, aggregates and semantic-consumption considerations.
• Data Quality & Testing profiling, rules, reconciliation, unit/integration testing, controls and defect analysis.
• Azure ecosystem Azure Data Lake Storage plus Azure integration, identity, monitoring and DevOps services.
• DevOps Git-based development, code reviews, CI/CD, environment promotion, release controls and automated deployment.
• Security & Governance least-privilege access, catalog, lineage, classification and row-level security.
• Advanced SQL and query optimization for large analytical datasets.
• Hands-on exposure to Databricks Lakeflow Spark Declarative Pipelines / Delta Live Tables (DLT) and ability to adopt evolving Databricks-native pipeline standards.
• Unity Catalog experience.
• Familiarity with enterprise data-governance/data-quality tools such as Informatica .
• Experience with HR data , preferably from Workday, Dayforce, PeopleSoft, recruiting, workforce management or talent systems.
• Power BI consumption patterns, star schemas, refresh requirements, aggregates and performance.
• Experience building accelerators, reusable engineering templates, metadata-driven pipelines or automated test frameworks .
• Databricks Data Engineer certification or equivalent cloud/data engineering credential.
• Experience working in a blended onshore/offshore Agile delivery model .

Key Responsibilities

• Designing and developing scalable batch data pipelines in Azure Databricks using PySpark and Spark SQL.
• Building and maintaining Bronze, Silver and Gold data structures, including certified fact and dimension data products.
• Translating business metric definitions, calculation rules, filters, granularity, history and refresh requirements into technical designs.
• Creating reusable HR-domain transformations covering areas such as retention, hiring, transfers, promotions, reviews, hours and scheduling .
• Implementing data-quality checks, reconciliation, exception handling, auditability, lineage and technical documentation.
• Optimizing Spark workloads, Delta tables, joins, partitions, file sizes and cluster usage for performance and cost.
• Applying enterprise security/governance standards, including Unity Catalog, permissions, sensitive-data handling and row-level access patterns.
• Coordinating with ingestion, data governance, modeling, DevOps and Power BI teams to establish clear data contracts and acceptance criteria.
• Contributing to CI/CD, source control, automated testing, deployment, operational monitoring and production-readiness activities.
• Participating in requirement workshops and communicating technical decisions to business, HR, analytics and engineering stakeholders.
• Supporting knowledge transfer and mentoring engineers/analysts.