Data Engineer

We are seeking a highly skilled Data Engineer to build and enhance an Azure-native third-party data enrichment platform using Databricks, Apache Spark, Snowflake, SQL, Python, and Azure Data Factory (ADF) . The role focuses on developing reliable, governed, scalable data pipelines while maintaining strong data quality, privacy, compliance, and cost efficiency. The ideal candidate will have strong hands-on experience with PySpark, SQL, ETL, Databricks, Spark troubleshooting, Snowflake, Azure Data Factory, and data engineering reliability patterns . You will work with large-volume third-party identity and attribute datasets supporting analytics, activation, research, and downstream Customer Data Platform (CDP) systems. Key Responsibilities Build and enhance data ingestion pipelines supporting large-scale batch and event-driven data paths. Integrate data from third-party enrichment vendors, digital platforms through Conversion API (CAPI), and Rewards/Promotions systems . Develop reliable ETL pipelines using PySpark, SQL, Databricks, Snowflake, Python, and Azure Data Factory . Implement strong data validation, idempotency, replay/backfill, deduplication, and auditability practices. Own pipeline monitoring, alerting, dashboarding, and operational readiness . Troubleshoot Spark failures using logs and perform root-cause analysis rather than simple reruns. Diagnose and optimize Spark performance issues involving shuffle, skew, partitioning, and workload performance . Apply privacy, compliance, and data governance requirements across pipelines and datasets. Support governance capabilities including Unity Catalog, data lineage, access controls, and PII/non-PII access management . Maintain documentation for tables, schemas, catalogs, and cluster usage . Design pipelines with cost efficiency and workload optimization in mind, including cluster sizing and compute/storage utilization. Balance cost, quality, performance, and SLA requirements when making engineering decisions. Raise and manage data quality escalations and contribute to evolving platform architecture. Collaborate effectively within a small, fast-moving data engineering team and take independent ownership of deliverables. Required Qualifications 4+ years of experience in Data Engineering or related technical roles. FAANG or Top Product Company background is mandatory. Strong hands-on experience with Data Engineering, SQL, Spark, ETL, and Python . 4+ years of experience with Databricks and Azure Data Factory (ADF) . Strong proficiency in PySpark and SQL , including hands-on coding rather than orchestration-only experience. Strong understanding of Apache Spark fundamentals , including partitioning, skew, shuffle optimization, and troubleshooting through Spark logs. Experience with Databricks notebooks/jobs, performance tuning, and medallion architecture patterns . Experience with Snowflake for data modeling, analytics, and data warehousing workloads. Experience with Azure Data Factory for data orchestration and Azure-native integrations. Strong understanding of data engineering reliability patterns including validation, idempotency, replay/backfills, deduplication, and auditability . Knowledge of data governance, lineage, access control, PII handling, and privacy/compliance requirements . Ability to work independently with strong ownership and problem-solving skills. In-person client interview is mandatory. Preferred Qualifications Experience with event-driven or streaming data ingestion . Experience with Delta Lake / Databricks patterns , including Delta Live Tables (DLT). Experience building config-driven export frameworks for multiple downstream consumers or vendors. Exposure to identity resolution concepts. Experience with Conversion API (CAPI) and marketing technology data signals. Experience implementing operational telemetry including dashboards, alerts, and SLA monitoring . Experience working with very large-scale datasets, including hundreds of millions of records per vendor . Experience with Unity Catalog and advanced Databricks governance capabilities . Strong understanding of data quality, scalability, compliance, governance, and cost optimization.

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers. #J-18808-Ljbffr BrickRed Systems