Senior Data Engineer

Job Summary

The Cloud Platform Manager is a senior technical leadership role responsible for designing, building, optimizing, and managing scalable enterprise data platforms within a cloud-based Azure ecosystem. This position plays a critical role in driving enterprise data engineering, analytics, machine learning enablement, and business intelligence initiatives through the development of modern Lakehouse architectures and high-performance data pipelines.

Key responsibilities include leading data engineering teams, architecting scalable Azure Databricks solutions, and implementing secure, high-quality enterprise data platforms that support analytics, reporting, and operational decision-making. The role combines hands-on technical expertise with leadership, governance, and strategic planning responsibilities to support enterprise-wide data initiatives.

Key Responsibilities

• Lead, mentor, and support teams of data engineers, analysts, and technical professionals while promoting development standards and best practices.
• Design, develop, deploy, and optimize scalable enterprise data platforms using Azure Databricks and Azure cloud technologies.
• Build and maintain robust batch and streaming ETL/ELT data pipelines using PySpark, Scala, SQL, and Azure-native tools.
• Architect modern Lakehouse environments utilizing Delta Lake and medallion architecture methodologies (Bronze, Silver, Gold layers).
• Collaborate with business stakeholders, data scientists, and technical teams to gather requirements and translate them into scalable technical solutions.
• Oversee data ingestion, transformation, integration, quality validation, governance, and storage optimization initiatives.
• Implement data governance frameworks, security controls, RBAC access models, lineage tracking, and compliance processes using Unity Catalog and related technologies.
• Integrate enterprise platforms with Azure Data Factory, Azure Data Lake Storage, Azure Synapse, Kafka, DevOps pipelines, and other cloud-native services.
• Optimize Spark workloads, Databricks clusters, query performance, indexing strategies, partitioning, and cost efficiency within Azure cloud environments.
• Support machine learning operations and model deployment initiatives utilizing MLflow and related MLOps tools.
• Develop and maintain scalable data models, schemas, and enterprise analytics frameworks.
• Participate in troubleshooting, production support, incident management, and continuous improvement initiatives for enterprise data environments.
• Ensure data platform reliability, scalability, performance, and operational excellence across cloud infrastructure.

Minimum Education & Experience Requirements

• Bachelor's degree in Computer Science, Information Technology, Engineering, Data Science, or a related technical field required.
• Minimum of 5–7 years of hands-on experience in data engineering, cloud architecture, or enterprise data platform development.
• Minimum of 2–4 years of direct experience working with Azure Databricks and Azure cloud data technologies.
• Prior experience leading or mentoring technical teams, including data engineers, analysts, or data scientists.
• Strong experience developing enterprise-scale ETL/ELT pipelines and distributed data processing solutions.
• Experience with modern cloud-based data platforms, big data technologies, and streaming architectures.
• Strong background in SQL, Python/PySpark, Scala, and enterprise data modeling principles.

Special Requirements

• Ability to work within a hybrid or remote enterprise technology environment.
• Availability to support production incidents or critical deployments outside standard business hours when necessary.
• Ability to work collaboratively across technical and non-technical business teams.
• Participation in strategic planning, architecture reviews, and enterprise governance initiatives as required.
• Relevant cloud and data engineering certifications are preferred.

Knowledge, Skills, and Abilities

• Deep expertise in Azure Databricks, Azure Data Factory, Azure Data Lake Storage, Azure Synapse, and Azure cloud infrastructure.
• Advanced knowledge of Delta Lake, Lakehouse architecture, and medallion data processing frameworks.
• Strong proficiency in Python, PySpark, SQL, Scala, and PowerShell.
• Expertise with Apache Spark runtime optimization and distributed data processing frameworks.
• Experience integrating real-time and batch data processing systems using Kafka, Spark Streaming, and related technologies.
• Knowledge of relational and NoSQL database technologies and data architecture principles.
• Familiarity with file formats such as Parquet, ORC, and Avro, including storage optimization techniques.
• Understanding of CI/CD processes, DevOps methodologies, and orchestration tools such as Airflow and Azure DevOps.
• Strong understanding of data governance, security, lineage, RBAC, compliance, and enterprise data management best practices.
• Excellent analytical, troubleshooting, and problem-solving abilities.
• Strong leadership, mentoring, organizational, and project coordination skills.
• Excellent written and verbal communication skills with the ability to communicate technical concepts to diverse audiences.
• Ability to manage multiple priorities in fast-paced enterprise environments.

Additional Desired Characteristics

• Microsoft Azure Data Engineer, Azure Solutions Architect, or Databricks certifications preferred.
• Experience supporting enterprise analytics, reporting, and machine learning initiatives.
• Familiarity with MLflow, MLOps practices, and AI/analytics platforms.
• Experience with large-scale enterprise data modernization initiatives.
• Exposure to legal, healthcare, financial services, or other highly regulated industries preferred.
• Strong understanding of enterprise cloud security, networking, and identity management concepts.
• Experience implementing enterprise-wide data governance and data quality initiatives.