Lead Data Engineer

Location: City of London, London Salary: £95000.00 - £110000.00 per annum Type: Permanent Location: London - Hybrid (3 days in the office) About the Client Our client is a leading specialist insurance provider, recognised for its innovative approach to technology & data. Operating globally, they are investing heavily in their modern data platform, leveraging cloud technologies to deliver scalable, high-quality data solutions that support underwriting, analytics & business decision-making. Role Overview As a Lead Data Engineer, you'll play a key role in designing & evolving a modern cloud-based data platform built on Databricks & Azure. You'll lead the development of scalable data pipelines using Python & PySpark, enabling high-quality data to power underwriting, analytics & business-critical decision-making across the organisation. Working closely with Product Owners, Architects & business stakeholders, you'll provide technical leadership throughout the full data lifecycle. Alongside remaining hands‑on, you'll mentor a team of Data Engineers, champion engineering best practices & help shape the future of the organisation's cloud data platform. Key Responsibilities: Lead & mentor a team of Data Engineers, supporting the successful delivery of data engineering projects from design through to production. Build & maintain cloud-native data solutions across the Azure ecosystem, including Azure Functions & related services. Collaborate with Product Owners, Architects & business stakeholders to define technical solutions & deliver against business priorities. Drive engineering best practices through code reviews, mentoring & continuous improvement initiatives. Identify opportunities to improve platform performance, data quality, scalability & operational efficiency. Champion automation, testing, CI/CD & modern engineering practices across the data engineering function. Key Requirements: Strong commercial experience building data engineering solutions using Python, PySpark & Databricks. Proven experience designing & optimising complex SQL queries & working with large-scale datasets. Strong experience across the Azure technology stack, including Azure Functions & cloud-native data services. Previous experience leading or mentoring Data Engineers within an Agile environment. Strong understanding of distributed data processing, modern data engineering practices & CI/CD. Experience building scalable, reliable & high-performing data platforms. Experience within Financial Services or Insurance is highly advantageous. Why Join: Opportunity to shape the future of a modern, cloud-native data platform built on Databricks & Azure. Hands‑on leadership role combining technical delivery with engineering strategy. Work with modern technologies including Python, PySpark, Databricks & the wider Azure ecosystem. Collaborative engineering culture focused on innovation, quality & continuous improvement. Hybrid working from a modern London office. Data & Application Notice We process personal data submitted as part of your application for recruitment purposes in line with UK GDPR. All applications are reviewed by a member of our team, we do not use automated decision‑making in our selection process. Your application will be assessed against the requirements of this role. By applying to this advert, you consent to receiving future relevant roles & industry news & insights from us. To opt out of this, please contact: View email address on click.appcast.io . #J-18808-Ljbffr Pioneer search