Lead Data Engineer

Role Overview

Do you have the right skills and experience for this role Read on to find out, and make your application.

The Lead Data Engineer will own the design, development, and maintenance of robust data architecture and pipelines leveraging cloud-based technologies, supporting strategic analytics, AI/ML initiatives, and data governance initiatives across the organization's enterprise-scale environment. Collaborating with cross-functional teams, this individual will ensure high-quality, secure, and performant data solutions that enable decision-making and operational excellence.

Key Responsibilities

• Design, build, test, deploy, and support scalable data pipelines utilizing orchestration tools (Airflow, Astronomer), data transformation frameworks (dbt), cloud data platforms (Snowflake), SQL, and Python.
• Develop and optimize cloud-based data architecture, including database schemas, data warehouses, security roles, workload isolation, cost management, and performance tuning within Snowflake.
• Implement medallion data architecture principles—raw/bronze, silver, and gold layers—for structured and semi-structured data management.
• Lead migration and modernization of legacy workflows to cloud orchestration platforms, establishing reusable patterns for dependency management, retries, backfills, alerting, recovery, and ensuring idempotency.
• Troubleshoot and maintain the full runtime environment of orchestration platforms, including schedulers, workers, executors, metadata stores, queues, and autoscaling components.
• Collaborate with cloud infrastructure teams to manage AWS services such as S3, IAM, EKS, ECS, Fargate, EC2, Lambda, Secrets Manager, KMS, CloudWatch, and networking components.
• Build secure, resilient integrations across APIs, SFTP, databases, cloud storage, vendor APIs, flat files, Excel files, and event-driven systems.
• Develop ingestion and processing pipelines for structured, semi-structured, and unstructured data, including OCR/image-based document extraction.
• Embed data quality testing, monitoring, lineage, reconciliation, and observability into data workflows to ensure reliability and compliance.
• Partner with Data Governance, Security, and Business teams to enforce data ownership, classification, access policies, retention, and auditing standards.
• Build and maintain CI/CD pipelines for orchestration workflows, transformations, and infrastructure code, ensuring automated testing, environment promotion, deployment, and rollback capabilities.
• Support production environments through on-call and incident management processes, conducting root cause analysis, stakeholder communication, and corrective actions.
• Establish reusable frameworks, engineering standards, and best practices to improve platform reliability, consistency, and delivery velocity.
• Lead technical workstreams, mentoring engineering team members through design reviews, code reviews, documentation, and troubleshooting.

Core Qualifications & Requirements

• Minimum of 5+ years in data engineering or software engineering roles with progressive responsibilities.
• Proven experience with cloud-based data platforms, including Snowflake, AWS cloud infrastructure, and data orchestration tools such as Airflow/Astronomer.
• Strong expertise in designing and implementing data architectures, including dimensional and medallion models.
• Hands-on experience migrating and maintaining production workflows in orchestration platforms.
• Deep understanding of cloud compute environments, including Kubernetes, containers, schedulers, workers, and autoscaling architectures.
• Proficiency in AWS services such as S3, IAM, EKS/ECS, Lambda, Secrets Manager, KMS, and CloudWatch for security, storage, and monitoring.
• Skilled in integrating complex data sources and systems, including APIs, SFTP, relational and NoSQL databases, cloud storage, and file-based workflows.
• Experience processing semi-structured and unstructured data, including OCR, document extraction, and image processing.
• Strong background in implementing data quality controls, automated testing, data lineage, and observability tools.
• Knowledge of data governance principles, access controls, data classification, retention policies, and audit requirements.
• Experience with CI/CD pipelines, Git workflows, and automated deployment practices for data solutions and infrastructure.
• Excellent troubleshooting skills across cloud infrastructure, data pipelines, application code, and security permissions. xhyhwjd
• Ability to lead technical initiatives and communicate effectively with both technical teams and business stakeholders.