Lead Data Engineer

Responsibilities : Lead the design, implementation, and optimisation of large-scale data processing solutions using Scala, Spark, SQL, and modern data platform technologies for a major workstream of the attribution platform. Lead the design and operation of trusted data processing pipelines that handle advertiser, customer, and measurement datasets within secure cloud environments (AWS, Google Cloud Platform, Azure) and approved data-sharing ecosystems. Collaborate with Product, Data Science, Security, Privacy, and Platform Engineering teams to deliver privacy-preserving attribution, measurement, forecasting, and analytics solutions. Own the design and operation of highly scalable batch and streaming data workflows using orchestration frameworks and cloud-native data services. Implement data classification, access controls, and privacy-preserving processing techniques to ensure sensitive datasets and identifiers are handled in accordance with security and compliance requirements. Drive the design and operation of clean-room and trusted data-sharing environments within your workstream, ensuring only approved aggregated or privacy-protected outputs are made available for downstream consumption. Build observability, monitoring, and operational tooling to ensure reliability, performance, and compliance of data processing platforms. Troubleshoot complex data platform, performance, and pipeline issues across distributed systems. Drive technical design, architecture decisions, and engineering best practices for a major workstream within the attribution platform, in partnership with Staff/Principal engineers. Mentor mid-level and senior engineers, lead design and code reviews, and provide technical leadership across your workstream. Continuously enhance OpenKyber's attribution, measurement, forecasting, and privacy-preserving analytics capabilities within your area of ownership. Strong written and verbal English communication skills are required. Good understanding of Agile/SCRUM methodologies and experience working within cross-functional product development teams.

Requirement : 8+ years of Data Engineering experience with strong Scala programming and extensive Apache Spark expertise for large-scale distributed data processing on AWS and/or Google Cloud Platform. Strong Python development skills for data pipelines, platform tooling, automation, and infrastructure modules. Advanced SQL skills across relational databases, cloud data warehouses, and lakehouse platforms; experience handling TB-scale datasets. Experience designing, building, and maintaining batch and streaming data pipelines. Strong understanding of data warehousing, dimensional modelling, data quality, partitioning, and performance optimisation. Experience with distributed data processing and modern lakehouse architectures (Databricks, Delta Lake, Apache Spark, or equivalent). Experience building and operating distributed data platforms at scale. Experience with workflow orchestration platforms such as Airflow, Databricks Workflows, AWS Step Functions, or equivalent DAG-based systems. Git or equivalent source control; unit, integration, and automated testing frameworks. Cloud-native development experience across AWS and/or Google Cloud Platform. Strong software engineering practices including CI/CD, code reviews, observability, and production support. Proven ability to lead technical design within a workstream, mentor mid-level and senior engineers, drive technical standards, and deliver within tight deadlines. Trusted environment execution skills (required). Experience designing and operating data pipelines within trusted data environments, clean rooms, secure data-sharing platforms, or equivalent privacy-preserving analytics environments. Experience working with sensitive datasets containing PII, customer identifiers, advertiser data, or regulated information. Experience implementing fine-grained access controls, data governance policies, and policy-based enforcement for sensitive datasets, including PII, PCI, and other regulated-data classification tiers. Familiarity with privacy-preserving data processing techniques including tokenisation, pseudonymisation, aggregation-before-export, and differential privacy concepts. Experience building or supporting clean-room, measurement, attribution, audience analytics, partner data-sharing, or privacy-preserving reporting solutions Experience with data lineage and governance tooling (Unity Catalog, AWS Glue Data Catalog, Apache Atlas, OpenLineage, or equivalent) for auditability and compliance. Understanding of trust boundaries, secure data-sharing patterns, and zero-trust data architecture principles. Experience documenting data contracts, data flows, lineage, and permitted movement of data between security zones and business domains. Experience with encryption, key management, and secure handling of sensitive data using cloud-native security services. Experience designing observability, monitoring, and alerting controls to detect anomalous data movement, policy violations, and potential data leakage events. Experience working in environments where only aggregated, anonymised, tokenised, or privacy-protected outputs may leave trusted processing environments. Strong understanding of cloud-native security and governance.

For applications and inquiries, contact:View email address on us.fitly.work