Azure Databricks Data Engineer

Data Engineer Position

Must Have Technical/Functional Skills

• Significant data engineering experience, including substantial hands-on Databricks development
• Azure Databricks - Strong hands-on experience building production-grade batch pipelines and lakehouse solutions.
• PySpark / Spark SQL - Advanced transformations, joins, window functions, reusable frameworks, debugging, and performance tuning.
• Delta Lake - Table design, schema evolution, merge/upsert patterns, optimization, data reliability, and history management.
• Medallion architecture - Practical delivery experience across source-aligned, conformed, and consumption-ready layers.
• Data modeling - Fact/dimension design, conformed dimensions, slowly changing dimensions, aggregates, and semantic-consumption considerations.
• Data quality and testing - Profiling, rule implementation, reconciliation, unit/integration tests, controls, and defect analysis.
• Azure ecosystem - Working knowledge of Azure Data Lake Storage and relevant Azure integration, identity, monitoring, and DevOps services.
• DevOps - Git-based development, code reviews, CI/CD, environment promotion, release controls, and automated deployment.
• Security/governance - Experience implementing least-privilege access and working with catalog, lineage, classification, and row-level security requirements.
• SQL - Advanced SQL development and query optimization for large analytical datasets.
• Hands-on exposure to Databricks Lakeflow Spark Declarative Pipelines or Delta Live Tables and the ability to adopt evolving Databricks-native pipeline standards.
• Experience with Unity Catalog; familiarity with enterprise data-governance and data-quality tooling such as Informatica.
• Experience with HR data from Workday, Dayforce, PeopleSoft, recruiting, workforce management, or talent systems.
• Understanding of Power BI consumption patterns, star schemas, refresh needs, aggregates, and performance requirements.
• Experience creating accelerators, reusable engineering templates, metadata-driven pipelines, or automated test frameworks.
• Databricks Data Engineer certification or equivalent cloud/data engineering credential.
• Experience in a blended onshore/offshore Agile delivery model.

Roles & Responsibilities

• Design and develop scalable batch data pipelines in Azure Databricks using PySpark and Spark SQL.
• Build and maintain bronze, silver, and gold data structures, including certified fact and dimension data products.
• Translate business metric definitions, calculation rules, filters, granularity, history, and refresh requirements into technical designs.
• Create reusable transformations for HR domains such as retention, hiring, transfers, promotions, reviews, hours, and scheduling.
• Implement data-quality checks, reconciliation, exception handling, auditability, lineage, and technical documentation.
• Optimize Spark workloads, Delta tables, joins, partitions, file sizes, and cluster usage for performance and cost.
• Apply enterprise security and governance standards, including Unity Catalog integration, permissions, sensitive-data handling, and support for row-level access patterns.
• Coordinate with ingestion, data governance, modeling, DevOps, and Power BI teams to define clear data contracts and acceptance criteria.
• Contribute to CI/CD, source control, automated testing, deployment, operational monitoring, and production-readiness activities.
• Participate in requirement workshops and explain technical choices clearly to business, HR, analytics, and engineering stakeholders.
• Support knowledge transfer and mentor participating engineers or analysts.

Salary Range-$110,000-$135,000 a year

Location: Saint Louis, MO

Function: TECHNOLOGY

Role: Engineer

ID: 435301

Skills: Azure

Salary Range: $110,000 - $135,000