Senior Data Engineer

Are you an experienced, passionate pioneer in technology who wants to work in a collaborative environment? As an experienced Senior Data Engineer you will share new ideas and collaborate on projects as a consultant without the extensive demands of travel at Deloitte under our Project Delivery Talent Model. The Project Delivery Model (PDM) is tailored specifically for long-term, onsite client service delivery. Responsibilities Communicate regularly with Engagement Managers (Directors), project team members, and representatives from various functional and/or technical teams, including escalating any matters that require additional attention and consideration from engagement management Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks Write and tune PySpark / Spark SQL notebooks for large-scale data transformation Architect end-to-end data solutions across dev > UAT > prod environments using Unity Catalog Lead and drive design discussions with client architects and other counterparts Collaborate with different teams on data contracts and schema agreements Lead design and optimization of high-volume data pipelines Define and enforce data engineering standards- naming conventions, partitioning strategies, cluster configurations, Spark tuning Drive performance optimization- AQE tuning, liquid clustering, broadcast joins, shuffle partition management Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies Conduct root-cause analysis on production incidents and implement permanent fixes Mentor junior and mid-level engineers through code reviews and pair programming Evaluate new technologies and recommend adoption (e.g., DABs, DLT, Auto Loader, Serverless Compute, event hubs) Qualifications Python, PySpark, Spark SQL, SQL Server Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor) Databricks (Delta Lake, Unity Catalog, Workflows) Apache Airflow Git / Azure DevOps Deep Spark internals (DAG optimization, spill analysis, skew handling) Delta Lake advanced features (time travel, deletion vectors, predictive I/O) Unity Catalog governance (row/column security, external locations, system tables) IaC – Terraform, Azure ARM templates Bachelor’s degree, preferably in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience Limited immigration sponsorship may be available Ability to travel 10%, on average, based on the work you do and the clients and industries/sectors you serve Benefits The wage range for this role is $95,000 to $150,000, reflecting factors such as skill set, experience, training, and other business needs. Information for applicants with a need for accommodation is available through Deloitte’s assistance for disabled applicants program. #J-18808-Ljbffr