Data Engineer

Responsibilities: Own the architecture and design of enterprise lakehouse-based data platforms, including Delta Lake, medallion architecture, and unified analytics layers. Serve as the senior technical authority on data platform design, guiding architecture, tooling, modelling, and engineering standards. Design and build ingestion, transformation, and orchestration pipelines using Python, SQL, PySpark, and workflow orchestration tools. Architect data mesh and data product strategies, defining domain ownership, data contracts, and self-service data consumption. Establish best practices across ingestion, transformation, modelling, quality, observability, governance, and data consumption. Drive integration with AI/ML capabilities, including feature engineering pipelines, vector data infrastructure for RAG, and ML lifecycle management. Lead complex data migration, platform modernisation, and consolidation initiatives. Evaluate emerging technologies such as Apache Iceberg, Delta Lake, Apache Hudi, Unity Catalogue, Delta Live Tables, and Mosaic AI to inform architectural decisions. Solve complex enterprise-scale data architecture challenges across multiple teams and platforms. Drive cross-functional collaboration across data engineering, analytics, AI/ML, platform engineering, security, and product teams. Mentor senior data engineers and analysts. Engage with client stakeholders on data strategy and technical roadmaps. Define and enforce governance, compliance (GDPR, HIPAA, SOC2), and responsible data management practices. Drive FinOps maturity through compute optimisation, storage lifecycle management, cluster governance, and cost optimisation. Requirements: 6-9 years of experience in data engineering, analytics engineering, or data architecture. Mandatory hands‑on expertise in Python, SQL, and enterprise lakehouse platforms. Strong knowledge of the modern data stack: dbt, Airflow/Dagster, Spark, Kafka, Fivetran/Airbyte, and cloud-native data services. Expertise in data modelling (Kimball, Data Vault, Activity Schema, OBT). Experience with cloud platforms (AWS, Azure, or GCP). Strong understanding of data governance, quality, lineage, cataloguing, and compliance. Experience supporting AI/ML workloads, including feature engineering, vector data, and model pipelines. Proven technical leadership and stakeholder management. Preferred Qualifications: Experience architecting enterprise data platforms supporting LLMs, RAG pipelines, and Agentic AI. Deep expertise in Delta Lake, Apache Iceberg, and Apache Hudi. Experience implementing data mesh and federated data governance. Experience with Kafka, Flink, and Spark Structured Streaming. Experience with Unity Catalogue, Delta Live Tables, MLflow, Mosaic AI, and workflow orchestration. Background in financial services, healthcare, SaaS, or enterprise consulting. Experience leading geographically distributed engineering teams. Must-Have Skills (Non-Negotiable): Lakehouse Platform Architecture: Hands‑on architecture and engineering experience on a production‑grade, enterprise‑scale lakehouse platform. Python: Strong professional proficiency for data engineering and pipeline development. SQL: Advanced proficiency for data modelling, transformation, and performance tuning. Candidates without demonstrable hands‑on experience in all three of the above will not be considered. #J-18808-Ljbffr Cogniify