Lead Data Engineer

We are partnered with an established Financial Services firm based in New York. They are hiring a Lead Data Engineer to lead the design and delivery of scalable data pipelines and cloud-based transformation frameworks across its Azure data platform. This is a hands‑on senior role for someone who can both build and lead — owning end‑to‑end pipeline architecture, mentoring on best practice, and working directly with business stakeholders to translate requirements into production‑grade data solutions. What you'll be doing: Design and build scalable ETL/ELT pipelines using Azure Data Factory, Databricks, and PySpark Architect and optimise data lake structures, including partitioning strategies for efficient querying at scale Integrate modern data tooling to support incremental updates, real‑time querying, and pipeline orchestration Own CI/CD for data solutions via Azure DevOps, covering ADF, Databricks, and Synapse deployments Partner with business owners and stakeholders on data governance, quality, and compliance Build dashboards and reporting layers in Power BI to support decision‑making Drive reliability improvements across pipeline monitoring, error handling, and retry logic What we're looking for: Strong hands‑on experience with Azure Databricks, Azure Data Factory, and Spark (Python/Scala) Proven experience architecting data lake and warehousing solutions at scale Experience with modern data tooling — DBT, Apache Airflow, Apache Hudi SQL and NoSQL proficiency, with experience across high‑volume, timeseries, or event‑driven data Comfortable operating in a stakeholder‑facing capacity — translating business needs into technical delivery #J-18808-Ljbffr