Data Engineer

Job Description We are seeking an experienced Data Engineer to design, build and maintain scalable data solutions on our Databricks platform. The successful candidate will be responsible for developing robust ETL/ELT processes, building high-performance data pipelines using PySpark, and integrating data from a variety of source systems into our Lakehouse environment. This role will play a key part in enabling data-driven decision making by delivering reliable, high-quality and governed data assets. The ideal candidate will have strong experience with Databricks, modern cloud-based data platforms, Data Lakehouse architecture, Medallion Architecture principles and enterprise-scale data engineering best practices. Key Accountabilities Design, develop and maintain scalable data pipelines within the Databricks platform. Build and optimise ETL/ELT processes using PySpark and Spark SQL. Implement and support Data Lakehouse and Medallion Architecture frameworks. Ingest, transform and integrate data from multiple internal and external source systems. Ensure data quality, accuracy, consistency and governance across data platforms. Monitor, troubleshoot and optimise data processing workloads and platform performance. Support the delivery of analytical, reporting and operational data solutions. Collaborate with the Product Owner, Business Analysts and business stakeholders to understand and deliver data requirements. Contribute to the development of data engineering standards, best practices and reusable frameworks. Support deployment, release management and CI/CD processes for data solutions. Qualifications Bachelor’s degree in Computer Science, Information Technology, Data Science or a related discipline. Databricks Certified Data Engineer Associate or Professional (preferred). Azure Fundamentals or Azure Data Fundamentals certifications (desirable). Relevant cloud, data engineering or analytics certifications are advantageous. Experience Minimum of 4+ years of hands‑on experience specifically within Databricks and PySpark environments. Broad background in Data Warehousing (10+ years preferred). Strong hands‑on experience building ETL/ELT pipelines using PySpark and Spark SQL. MSX is an equal opportunities employer and encourages applications from suitably qualified and eligible candidates regardless of sex, race, disability, neurodiversity or other personal characteristics and backgrounds, age, sexual orientation, gender reassignment, religion or belief, or marital and parental status. As users of the Disability Confident scheme, we interview all disabled applicants who meet the minimum criteria for the vacancy. #J-18808-Ljbffr