Sr. Big Data Engineer

Staff Data EngineerHybrid at least 2 days per week in office in Chicago, ILOur client seeks a Staff Data Engineer to set technical direction for large-scale data processing in a privacy-preserving attribution platform. The role will architect and operate secure pipelines across cloud environments, collaborate with cross-functional teams, implement governance and access controls, and drive architecture for batch and streaming workflows. The engineer will build observability and operational tooling, troubleshoot distributed systems, mentor senior engineers, and evolve attribution, measurement, forecasting, and analytics capabilities with strong security and compliance alignment.Responsibilities:Set technical direction for large-scale data processing solutions using Scala, Spark, SQL, and modern data platforms across the attribution ecosystem.Architect and operate trusted pipelines handling advertiser, customer, and measurement datasets within secure cloud environments and approved data-sharing ecosystems.Collaborate with Product, Data Science, Security, Privacy, and Platform Engineering to deliver privacy-preserving attribution, measurement, forecasting, and analytics solutions.Define architecture for batch and streaming workflows, including orchestration frameworks and cloud-native services.Implement data classification, access controls, and privacy-preserving processing techniques to meet security and compliance requirements.Establish architectural direction for clean-room and trusted data-sharing environments with approved aggregated or privacy-protected outputs.Build observability, monitoring, and operational tooling to ensure reliability, performance, and compliance.Troubleshoot complex platform, performance, and pipeline issues across distributed systems.Drive architecture decisions, engineering best practices, and operational excellence across attribution and adjacent data products.Mentor and grow senior and lead engineers and provide technical leadership across teams and product areas.Design and implement pipelines within trusted environments across AWS, GCP, Azure, and on-premises infrastructure.Implement access controls, data classification policies, lineage tracking, and governance controls for sensitive data.Define and maintain data handling standards with Security, Privacy, and Compliance stakeholders.Enforce privacy-preserving principles to ensure only aggregated, anonymized, tokenized, or otherwise approved outputs leave trusted environments.Build monitoring and alerting to detect anomalous movement, policy violations, and potential leakage.Apply privacy-preserving computation techniques, including aggregation-before-export, pseudonymization and tokenization, differential privacy concepts, and privacy-aware reporting.Implement encryption, key management, and secure data handling using cloud-native services.Document trust boundaries, data contracts, lineage, and permitted data movement between zones.Support audits, compliance requirements, governance reviews, and secure data-sharing initiatives.Participate in architecture and design reviews for new data products to incorporate governance, privacy, lineage, and trust-boundary requirements.Contribute to engineering standards and best practices for secure data processing and privacy-preserving analytics.Experience Requirements:10+ years of data engineering with deep Scala and extensive Apache Spark for large-scale processing on AWS and/or GCP.Strong Python for pipelines, tooling, automation, and infrastructure modules.Advanced SQL across relational, cloud warehouses, and lakehouse platforms with TB-scale datasets.Designing, building, and maintaining batch and streaming pipelines.Strong understanding of warehousing, dimensional modeling, data quality, partitioning, and performance optimization.Experience with lakehouse architectures such as Databricks, Delta Lake, or equivalent.Operating distributed data platforms at scale.Workflow orchestration with Airflow, Databricks Workflows, AWS Step Functions, or equivalent.Source control and testing practices including unit, integration, and automation.Cloud-native development on AWS and/or GCP.Strong engineering practices including CI/CD, code reviews, observability, and production support.Proven ability to set technical direction across teams and mentor senior engineers.Trusted environment execution: clean rooms and secure data-sharing platforms, handling PII and regulated data, fine-grained access controls, and policy-based enforcement.Working knowledge of data classification frameworks including PII, PCI, regulated data, and sensitivity-tier models.Familiarity with tokenization, pseudonymization, aggregation-before-export, and differential privacy concepts.Experience with clean-room, measurement, attribution, audience analytics, partner data-sharing, or privacy-preserving reporting.Lineage and governance tooling such as Unity Catalog, AWS Glue Data Catalog, Apache Atlas, or OpenLineage.Understanding of trust boundaries, secure sharing patterns, and zero-trust data architecture principles.Documenting data contracts, flows, lineage, and permitted movement between security zones and domains.Encryption, key management, and secure handling using cloud-native security services.Designing observability and alerting controls to detect anomalous movement, policy violations, and potential leakage.Experience in environments where only aggregated, anonymized, tokenized, or privacy-protected outputs may exit trusted processing environments.Strong understanding of cloud-native security and governance.Good to have: Databricks ecosystem, AWS Clean Rooms, advertising measurement and attribution platforms, collaboration with Security and Compliance in regulated settings, ELK, Grafana, OpenTelemetry, Docker, Kubernetes, and security architecture or threat modeling.Strong written and verbal English communication skills.Good understanding of Agile and SCRUM within cross-functional teams.