Senior Data Engineer

Senior Data Engineer (Machine Learning Enablement)

Location: Atlanta, GA (Onsite occasionally to Midtown Atlanta client site)

POSITION OVERVIEW

We are seeking a Senior Data Engineer with machine learning capabilities to build the scalable data foundation powering our ML models. You will lead the design of well-governed, cloud-native data platforms on Azure, bringing external retail and third-party data into a medallion architecture aligned to client master data. In this role, you will collaborate closely with Data Science and ML teams to set technical direction for data preparation and delivery.

DAY-TO-DAY RESPONSIBILITIES

• Onboard and align external retailer and third-party datasets into a medallion architecture (bronze, silver, gold).
• Profile retailer feeds to define partitioning strategies and address data gaps via imputation, mapping, or source blending.
• Define and validate join keys against product, outlet, and customer dimension tables, resolving unmatched or duplicate keys.
• Build and tune ETL/ELT data pipelines using Azure Data Factory, Azure Databricks, Azure Functions, and Synapse.
• Execute large-scale data migrations-including multi-terabyte transactional volumes-with minimal to zero downtime.
• Establish enterprise data quality controls, governance frameworks, and feature-ready models for recurring business cycles.
• Partner with Data Science/ML teams to productionize models and maintain clear documentation and version control in GitHub.

REQUIRED SKILLS

• Data Engineering Lead/Architect Experience: 7+ years building and managing production data platforms.
• Azure Cloud Stack: 7+ years of hands-on experience with Azure Data Factory, Databricks, Functions, and Synapse.
• Python & SQL: 7+ years of strong experience using PySpark or pandas for large-scale data transformation.
• Architecture & Modeling: 7+ years of practical application of medallion architecture, dimensional modeling, and master data joins.

ADDITIONAL DESIRED SKILLS

• Multi-Cloud Experience: AWS or GCP knowledge.
• CRM Integration: Hands-on Salesforce integration (data sync, APIs, or migrations).
• Domain Knowledge: Consumer goods or retail datasets (POS, syndicated data).
• Machine Learning: Experience supporting ML feature pipelines and model productionization.

CANDIDATE PROFILE

The ideal candidate is an architectural-minded Senior Data Engineer who thrives at the intersection of enterprise data platforms and machine learning enablement. They possess strong technical leadership, deep expertise in Azure data ecosystems, and the communication skills necessary to bridge gaps between technical data science teams and business stakeholders.

Job Responsibilities

• Design, build, and optimize scalable ETL/ELT data pipelines using Azure Data Factory, Databricks, Functions, and Synapse.
• Profile and onboard external third-party/retail datasets into a medallion architecture (bronze, silver, gold) aligned with master dimension tables.
• Execute large-scale data migrations and refactor complex semi-structured source data (e.g., XML) into analysis-ready structures.
• Establish enterprise-wide data governance frameworks, quality controls, and feature-ready models to support recurring business cycles.
• Partner with Data Science teams to productionize ML models while enforcing Git code review and documentation standards.