Lead Data Engineer

Data Engineering-Big Data Engineering

As a technical lead, you will guide engineering teams, establish best practices, and ensure successful delivery across multiple workstreams while enabling data-driven decision-making across the enterprise.

Responsibilities

Data Engineering & Platform Development

• Design, develop, and maintain scalable data pipelines and data products supporting analytics, reporting, AI, and operational use cases.
• Build and optimize ETL/ELT frameworks for large-scale data ingestion, transformation, validation, and consumption.
• Develop and manage cloud-native data platforms leveraging Snowflake, AWS, Apache Airflow and modern data architectures.
• Create scalable data models, data marts, semantic layers, and curated datasets that support enterprise analytics initiatives.
• Optimize SQL workloads, transformation logic, and query performance to improve scalability and cost efficiency.
• Establish reusable engineering frameworks, accelerators, and best practices to improve delivery consistency across projects.
• Ensure high standards of data quality, reliability, governance, and observability throughout the data lifecycle.
• Develop and maintain Snowflake-based data ecosystems, leveraging advanced features for performance optimization and data sharing.
• Build and orchestrate data workflows using Airflow and other workflow scheduling platforms.
• Collaborate directly with client stakeholders to gather requirements, define roadmaps, and develop scalable technical solutions.
• Present solution designs, technical recommendations, and project updates to both technical and business audiences.
• Prepare and maintain comprehensive project documentation, technical specifications, architecture diagrams, and operational runbooks.

Qualifications

Required Qualifications

• 4+ years of experience in Data Engineering, Big Data Engineering, or Cloud Data Platform development.
• Bachelor's or Master's degree in Computer Science, Engineering, Analytics, Mathematics, Information Systems, or related disciplines.
• Strong hands-on expertise in SQL, Python, and PySpark.
• Extensive experience working with Snowflake, Databricks, or similar cloud-native data platforms.
• Proven experience building and supporting large-scale ETL/ELT data pipelines.
• Strong understanding of data warehousing concepts, dimensional modeling, and modern Lakehouse architectures.
• Experience implementing Medallion Architecture and enterprise-grade data modeling practices.
• Hands-on experience with workflow orchestration tools such as Apache Airflow or equivalent scheduling frameworks.
• Experience working with cloud ecosystems including AWS, Azure, or GCP.
• Strong knowledge of performance tuning, optimization, monitoring, and operational support for data platforms.
• Demonstrated experience leading engineering teams and coordinating with client and internal stakeholders.
• Excellent analytical, problem-solving, communication, and stakeholder management skills.
• Ability to work independently and lead complex initiatives in fast-paced consulting environments.

Preferred Qualifications

• Experience with streaming and real-time data processing frameworks.
• Familiarity with DataOps, CI/CD, Infrastructure as Code, and DevOps practices.
• Experience with data governance, data quality frameworks, and metadata management.
• Exposure to AI/ML data pipelines and feature engineering workflows.
• Experience with visualization tools such as Tableau, Power BI, or Looker.
• Hands-on experience with Big Data technologies including Spark, Hadoop, Hive, HBase, Kafka, or related platforms.
• Consulting or client-facing delivery experience in enterprise-scale environments.

Required Skills

• Data Architecture
• Data Strategy
• ETL Design
• Interpersonal Dynamics with Coworkers
• Resource Management
• Results Orientation
• Time Management Skills
• Working under Pressure
• Writing Communication Skills