Senior Data Engineer

Job Description

Job Summary

\n

The Cloud Platform Manager is a senior technical leadership role responsible for designing, building, optimizing, and managing scalable enterprise data platforms within a cloud-based Azure ecosystem. This position plays a critical role in driving enterprise data engineering, analytics, machine learning enablement, and business intelligence initiatives through the development of modern Lakehouse architectures and high-performance data pipelines.

\n

Key responsibilities include leading data engineering teams, architecting scalable Azure Databricks solutions, and implementing secure, high-quality enterprise data platforms that support analytics, reporting, and operational decision-making. The role combines hands-on technical expertise with leadership, governance, and strategic planning responsibilities to support enterprise-wide data initiatives.

\n

\n

Key Responsibilities

\n

\n
• Lead, mentor, and support teams of data engineers, analysts, and technical professionals while promoting development standards and best practices.
\n
• Design, develop, deploy, and optimize scalable enterprise data platforms using Azure Databricks and Azure cloud technologies.
\n
• Build and maintain robust batch and streaming ETL/ELT data pipelines using PySpark, Scala, SQL, and Azure-native tools.
\n
• Architect modern Lakehouse environments utilizing Delta Lake and medallion architecture methodologies (Bronze, Silver, Gold layers).
\n
• Collaborate with business stakeholders, data scientists, and technical teams to gather requirements and translate them into scalable technical solutions.
\n
• Oversee data ingestion, transformation, integration, quality validation, governance, and storage optimization initiatives.
\n
• Implement data governance frameworks, security controls, RBAC access models, lineage tracking, and compliance processes using Unity Catalog and related technologies.
\n
• Integrate enterprise platforms with Azure Data Factory, Azure Data Lake Storage, Azure Synapse, Kafka, DevOps pipelines, and other cloud-native services.
\n
• Optimize Spark workloads, Databricks clusters, query performance, indexing strategies, partitioning, and cost efficiency within Azure cloud environments.
\n
• Support machine learning operations and model deployment initiatives utilizing MLflow and related MLOps tools.
\n
• Develop and maintain scalable data models, schemas, and enterprise analytics frameworks.
\n
• Participate in troubleshooting, production support, incident management, and continuous improvement initiatives for enterprise data environments.
\n
• Ensure data platform reliability, scalability, performance, and operational excellence across cloud infrastructure.
\n

\n

\n

Minimum Education & Experience Requirements

\n

\n
• Bachelor’s degree in Computer Science, Information Technology, Engineering, Data Science, or a related technical field required.
\n
• Minimum of 5–7 years of hands-on experience in data engineering, cloud architecture, or enterprise data platform development.
\n
• Minimum of 2–4 years of direct experience working with Azure Databricks and Azure cloud data technologies.
\n
• Prior experience leading or mentoring technical teams, including data engineers, analysts, or data scientists.
\n
• Strong experience developing enterprise-scale ETL/ELT pipelines and distributed data processing solutions.
\n
• Experience with modern cloud-based data platforms, big data technologies, and streaming architectures.
\n
• Strong background in SQL, Python/PySpark, Scala, and enterprise data modeling principles.
\n

\n

\n

Special Requirements

\n

\n
• Ability to work within a hybrid or remote enterprise technology environment.
\n
• Availability to support production incidents or critical deployments outside standard business hours when necessary.
\n
• Ability to work collaboratively across technical and non-technical business teams.
\n
• Participation in strategic planning, architecture reviews, and enterprise governance initiatives as required.
\n
• Relevant cloud and data engineering certifications are preferred.
\n

\n

\n

Knowledge, Skills, and Abilities

\n

\n
• Deep expertise in Azure Databricks, Azure Data Factory, Azure Data Lake Storage, Azure Synapse, and Azure cloud infrastructure.
\n
• Advanced knowledge of Delta Lake, Lakehouse architecture, and medallion data processing frameworks.
\n
• Strong proficiency in Python, PySpark, SQL, Scala, and PowerShell.
\n
• Expertise with Apache Spark runtime optimization and distributed data processing frameworks.
\n
• Experience integrating real-time and batch data processing systems using Kafka, Spark Streaming, and related technologies.
\n
• Knowledge of relational and NoSQL database technologies and data architecture principles.
\n
• Familiarity with file formats such as Parquet, ORC, and Avro, including storage optimization techniques.
\n
• Understanding of CI/CD processes, DevOps methodologies, and orchestration tools such as Airflow and Azure DevOps.
\n
• Strong understanding of data governance, security, lineage, RBAC, compliance, and enterprise data management best practices.
\n
• Excellent analytical, troubleshooting, and problem-solving abilities.
\n
• Strong leadership, mentoring, organizational, and project coordination skills.
\n
• Excellent written and verbal communication skills with the ability to communicate technical concepts to diverse audiences.
\n
• Ability to manage multiple priorities in fast-paced enterprise environments.
\n

\n

\n

Additional Desired Characteristics

\n

\n
• Microsoft Azure Data Engineer, Azure Solutions Architect, or Databricks certifications preferred.
\n
• Experience supporting enterprise analytics, reporting, and machine learning initiatives.
\n
• Familiarity with MLflow, MLOps practices, and AI/analytics platforms.
\n
• Experience with large-scale enterprise data modernization initiatives.
\n
• Exposure to legal, healthcare, financial services, or other highly regulated industries preferred.
\n
• Strong understanding of enterprise cloud security, networking, and identity management concepts.
\n
• Experience implementing enterprise-wide data governance and data quality initiatives.
\n

\n

\n