Senior Data Engineer - AWS / Databricks

About Dateonic & the project Big Data and AI are in our DNA, with Databricks at the core of our innovation. Founded by engineers and architects passionate about tackling complex data challenges, Dateonic combines industry-leading technical expertise with a product-focused approach to turn raw data into actionable intelligence. Our culture is driven by three core values: Passion, Community, and Commitment. We love tough problems, support each other through close collaboration, and deliver what we promise without taking shortcuts. In this outsourcing role, you will represent Dateonic on a long-term initiative with a global leader in experience innovation. Our partner operates at the intersection of data, AI, creativity, and technology to build modern digital platforms for world-renowned enterprise brands. Joining this project allows you to work on high-impact, global architecture while enjoying the support, engineering community, and technical mentorship of the Dateonic team. The role We are looking for an experienced Senior Data Engineer to design, build, and optimise modern, cloud-based data platforms powering analytics, AI, and data products. Beyond technical delivery, we value professionals who are genuinely curious about the business context behind the numbers, applying practical analysis and experience to help shape requirements rather than simply implementing them as given. Our project work is centred on AWS with Databricks, alongside growing demand for GCP, including cloud migration programmes. Azure and Fabric experience is also valued. We look for T‑shaped engineers with deep expertise in one primary cloud who are adaptable and eager to broaden their multi‑cloud capabilities. Responsibilities Design, implement, and optimise scalable batch, streaming, and near‑real‑time data platforms using Apache Spark and Delta Lake, primarily on AWS and Databricks (with GCP or Azure/Fabric where project streams require). Build robust ingestion, transformation, and curation workflows for structured and unstructured data, applying lakehouse patterns and medallion layering (bronze, silver, gold). Support cloud-to-cloud migrations, including re-platforming legacy pipelines, data models, and storage layers. Deliver production‑grade, model‑ready datasets and build data pipelines for machine learning, causal modelling, and GenAI use cases (such as LLMs, vector search, RAG, and agent‑based architectures). Design logical and physical data models for analytical and operational use cases, orchestrating automated workflows with Databricks Workflows, Airflow, dbt, or cloud‑native equivalents. Enforce enterprise data governance, lineage, quality frameworks, and granular access controls (RBAC/ABAC) using Unity Catalog, AWS Lake Formation, or equivalent cloud tools. Implement robust platform observability, monitoring data freshness, pipeline reliability, cost efficiency, and SLA compliance. Build serving layers (APIs, feature inputs, analytical endpoints) and partner closely with data scientists, ML engineers, and business stakeholders to challenge requirements and surface better architectural solutions. Must‑have requirements Strong hands‑on experience with Apache Spark, Delta Lake, and production‑grade batch and streaming pipeline development. Advanced programming skills in Python and SQL with a solid grasp of data modelling, data quality, and governance principles. Proven, demonstrable hands‑on experience with the Databricks Lakehouse Platform, ideally on AWS (utilising services such as S3, Glue, EMR, Redshift, Kinesis, Lambda). Deep technical expertise in at least one major cloud (AWS, GCP, or Azure/Fabric) paired with an openness to work across clouds (candidates with strong Azure/Fabric experience are welcome provided they want to expand multi‑cloud skills). Solid understanding of distributed systems, lakehouse architectures, scalability, and system performance tuning. A proactive, consultative mindset: comfortable asking probing questions, challenging assumptions, and collaborating effectively in cross‑functional agile teams. Nice to have Hands‑on experience across multiple clouds or direct experience in cloud‑to‑cloud migrations (for example, Azure to GCP). Practical experience with Google Cloud Platform services (BigQuery, Dataflow, Pub/Sub, Cloud Storage, Dataproc). Familiarity with Snowflake. Hands‑on exposure to GenAI data pipelines, vector databases, and LLM data preparation. Experience with Infrastructure as Code (Terraform, CloudFormation) and CI/CD pipelines for automated data deployments. Knowledge of event streaming technologies (such as Apache Kafka) or ML/experimentation tooling (such as Databricks MLflow). #J-18808-Ljbffr