Senior Data Engineer

Job Title

Data Engineering & Platform Medallion Pipeline Ownership

Job Description

This is a remote position. Data Engineering & Platform Medallion pipeline ownership — Build, maintain, and optimize Bronze, Silver, and Gold layer pipelines in Microsoft Fabric using ADF (Azure Data Factory) and PySpark notebooks. Own the full ingestion-to-consumption chain for all source systems including Acumatica ERP, HubSpot CRM, AutoStore, RingCentral, website analytics, and marketplace channel data (Net32, Amazon, eBay, and others). Event-driven ingestion — Implement and maintain event-triggered ADF pipelines using Acumatica SaaS push notifications routed through Azure Service Bus and Azure Function App receivers. Build OData watermark fallback pipelines with LastModifiedDateTime-based incremental load, pagination handling, and session concurrency management. CDC and incremental load — Design and manage Change Data Capture strategies including CDF (Change Data Feed) versioning in the watermark registry, last-processed version tracking, and initial vs. incremental load logic. Ensure Silver batching strategy (currently 37 tables across 3 batches) runs within SLA and scale intelligently as new sources are onboarded. Bronze-to-Gold transformation — Develop and maintain Silver transformation notebooks including deduplication, null handling, type casting, and DQ rule application. Build and optimize Gold KPI layer notebooks following a star schema — fact and dimension tables — with orchestration notebooks managing execution sequence and dependency management. Schema management and evolution — Validate Bronze Delta table schemas against source system payloads. Implement schema evolution handling via the Delta transaction log. Maintain the source_object_config and watermark_registry control tables that govern what is ingested and how. Data quality enforcement — Implement and maintain DQ rules (DQ-001 through DQ-020) covering completeness, uniqueness, validity, consistency, timeliness, and referential integrity across all Silver layer tables for both Acumatica and HubSpot sources. Route failed records to the DQ failed records table and ensure clean records only proceed to Gold. CI/CD and environment management — Manage the Git promotion flow from dev to production across Bronze, Silver, and Gold layers using Azure DevOps. Maintain environment parity between dev-platform and prod environments including library versions, environment variables, and connection configurations. Follow the pipeline-based promotion workflow — never direct workspace-to-Git sync. Platform observability — Instrument notebooks with the custom SDK logging framework (send_log, send_metric, send_error) using Azure Log Analytics as the destination. Ensure all production notebooks call logging_setup at the start of each run. Monitor pipeline health dashboards for error rates, duration anomalies, and row count deviations. Pecan AI and ML model data feeds — Maintain the data pipeline feeds that power Pecan AI forecasting and recommendation models including demand forecasting, Vegas inventory forecasting, customer churn propensity, and next-best-product models. Ensure Fabric SQL endpoint bidirectional integration with Pecan is healthy and model retraining pipelines are reliable.