Senior Data Platform Engineer

Overview of the Function

We are seeking an experienced and forward-thinking Senior Data Platform Engineer to manage optimize and evolve the cloud-native platforms used by the Data Office. The role combines deep data engineering capability with platform engineering accountability across Microsoft Azure Databricks Unity Catalog integration and ingestion services Collibra and Immuta.

The successful candidate will ensure that the Data Office platform estate is secure reliable scalable observable and cost-effective. This includes optimizing performance workloads storage compute access and platform configurations while also designing and delivering robust data pipelines and lakehouse solutions that support analytics business intelligence AI and machine learning.

This is a hands-on role working across engineering architecture governance security and operations to provide a trusted enterprise data platform and a strong developer and consumer experience.

Role Summary / Purpose of Job

The Senior Data Platform Engineer will be responsible for the day-to-day engineering and technical management of the Data Office platform ecosystem. The role will maintain platform health availability security performance and cost efficiency while supporting platform upgrades configuration automation monitoring access management and incident resolution.

In parallel the role will deliver senior-level data engineering outcomes across ingestion integration transformation orchestration governance and data access. The engineer will work with data architects engineers analysts governance teams security teams and business stakeholders to ensure that data products are reliable discoverable controlled and fit for purpose.

Key Responsibilities and Deliverables

• Manage the Data Office platform estate across Azure Databricks Unity Catalog integration and ingestion services Collibra and Immuta ensuring operational reliability and alignment with enterprise standards.
• Monitor and optimize platform performance workloads clusters SQL warehouses jobs storage and compute to improve reliability scalability user experience and cost efficiency.
• Own capacity and workload management including resource allocation concurrency job scheduling autoscaling workload isolation and the assessment of serverless and cluster-based compute options.
• Administer platform configuration workspaces environments service connections secrets policies and technical integrations across development test and production.
• Manage data access controls and entitlements across Azure Databricks Unity Catalog Collibra and Immuta applying least-privilege role-based attribute-based and policy-based access principles.
• Implement and maintain Unity Catalog structures including catalogs schemas permissions data objects lineage and controlled data-sharing patterns.
• Support Collibra integration and adoption for cataloging metadata business glossaries lineage ownership stewardship and governance workflows.
• Design build and maintain secure scalable lakehouse solutions using Azure Data Lake Storage Databricks Delta Lake Spark PySpark Python and SQL.
• Develop and optimize batch and streaming ingestion integration ETL and ELT pipelines using Azure Data Factory Databricks Workflows APIs event-based services and appropriate orchestration tools.
• Engineer reusable integration and ingestion patterns for structured semi-structured and unstructured data from enterprise applications files databases APIs and event streams.
• Implement observability across platforms and pipelines including monitoring logging alerting performance metrics usage reporting cost reporting failure analysis and operational dashboards.
• Lead technical incident investigation root-cause analysis remediation and problem management for platform workload access and data pipeline issues.
• Automate platform deployment and configuration through infrastructure as code CI/CD testing release management and version control using tools such as Terraform Git Azure DevOps or GitHub Actions.
• Implement data quality reconciliation validation exception handling recovery and audit controls to improve reliability and trust in enterprise data products.
• Plan and execute platform upgrades feature adoption migrations configuration changes resilience testing backup and recovery activities and lifecycle management.
• Maintain technical standards runbooks architecture diagrams support procedures data dictionaries operating documentation and knowledge articles.
• Provide technical leadership coaching design review and best-practice guidance to data engineers and other platform users.
• Collaborate with architecture cloud operations cyber security risk governance analytics and business teams to translate requirements into secure and sustainable engineering solutions.

Key Requirements

Experience and Background

• 510 years of hands-on experience in data engineering data platform engineering cloud engineering or a closely related field.
• Proven experience managing and supporting enterprise data platforms in production including performance workloads storage compute access availability and cost.
• Proven experience designing and implementing cloud data lake or lakehouse architectures at scale.
• Experience operating in a Microsoft Azure enterprise environment and working with cross-functional platform security governance and engineering teams.
• Experience working in a regulated environment preferably legal or financial services with an understanding of security privacy audit and compliance obligations.

Core Platform and Engineering Skills

• Advanced hands-on experience with Databricks including Spark PySpark Delta Lake SQL Workflows clusters SQL warehouses jobs performance tuning and cost optimization.
• Hands-on experience with Unity Catalog including catalogs schemas permissions lineage secure data access and governance administration.
• Strong experience with Microsoft Azure services including Azure Data Lake Storage Gen2 Azure Data Factory Azure Functions Logic Apps Key Vault networking identity monitoring and related services.
• Strong Python and SQL skills with the ability to develop review troubleshoot and optimize production-grade code and queries.
• Experience designing and supporting integration and ingestion patterns across databases enterprise applications files APIs and streaming or event-based sources.
• Experience with Collibra or a comparable data catalog and governance platform including metadata lineage stewardship ownership and workflow integration.
• Experience with Immuta or a comparable data security platform including policy-based access sensitive data controls auditability and integration with Databricks.
• Knowledge of identity access management role-based access control attribute-based access control service principals managed identities secrets and least-privilege design.
• Experience with platform monitoring observability capacity management incident response root-cause analysis resilience backup and recovery.
• Experience with CI/CD infrastructure as code source control automated testing and release management using Terraform Git Azure DevOps GitHub Actions or equivalent tooling.

Desirable Skills

• Experience with dbt Apache Airflow Azure Event Hubs Spark Structured Streaming or comparable transformation and orchestration technologies.
• Experience with Azure Synapse Analytics Microsoft Purview containerization Kubernetes or enterprise API management.
• Knowledge of FinOps practices for data platforms including attribution budgeting forecasting chargeback or showback and cost-control policies.
• Understanding of data quality metadata management lineage retention classification privacy and records-management controls.
• Familiarity with agile delivery and IT service management practices including change incident problem and release management.

Professional Skills

• Strong analytical problem-solving troubleshooting and root-cause analysis skills.
• Ability to balance platform stability and control with delivery speed usability and innovation.
• Clear verbal and written communication skills with the ability to work effectively across technical and non-technical teams.
• Ability to prioritize operational work engineering delivery platform improvements and technical debt.
• Proactive ownership sound judgement attention to detail and commitment to continuous improvement and knowledge sharing.

EEO Statement
Freshfields US LLP is proud to be an equal employment employer. Our policies and practices will be free from unlawful discrimination based upon race color ethnicity religion creed sex (including pregnancy childbirth or related medical conditions) national origin citizenship immigration status ancestry age marital status protected veteran status military service disability medical condition genetic information sexual orientation gender identity or any basis prohibited under federal state or local law. We strive to promote an atmosphere that encourages equal opportunities and prohibits discriminatory practices including sexual harassment.

Required Experience:

Senior IC