Lead Data Engineer

Job Description: The Lead Data Engineer will lead and support the design, development, and delivery of complex data engineering and AI-ready data solutions across the enterprise data ecosystem. This role will work closely with Product Engineers, Data Engineers, business-facing leads, architects, Data Scientists, AI Engineers, and cross-functional stakeholders to translate business needs into scalable data and AI solutions. The successful candidate will bring strong hands-on experience with Snowflake data warehouse, ETL/ELT tools such as Nexla, workflow orchestration platforms such as Apache Airflow, and modern AI-powered development tools including Microsoft Copilot, Claude, and other GenAI platforms. The ideal candidate will understand how to design data foundations that support analytical workloads, machine learning, large language models (LLMs), Retrieval-Augmented Generation (RAG), and Agentic AI frameworks. This role is expected to help manage team priorities within Agile sprints, coordinate effectively across onsite, offsite, offshore, and cross-team members, and drive the adoption of AI-assisted engineering practices that improve productivity, quality, and delivery speed.

Position Responsibilities: Lead the analysis, design, development, and implementation of scalable data solutions within the Enterprise Data Warehouse and modern data platform. Partner with Product Engineers, Data Engineers, business-facing leads, architects, Data Scientists, AI Engineers, and stakeholders to understand requirements, clarify priorities, and deliver business-aligned data and AI solutions. Design, build, optimize, and support ETL/ELT pipelines using Snowflake, Nexla, SQL, Python, and related data integration technologies. Develop and manage Apache Airflow jobs, DAGs, and schedules to ensure reliable, observable, and timely data processing across platforms. Design and implement AI-ready data architectures that support analytics, machine learning, generative AI, RAG, vector databases, and Agentic AI solutions. Leverage AI-assisted development tools such as Microsoft Copilot, Claude, Cursor, and similar platforms to accelerate development, improve code quality, automate documentation, and enhance engineering productivity. Collaborate with AI/ML teams to define data ingestion, transformation, governance, and serving patterns required for AI model training and inference workloads. Support the implementation of metadata management, data lineage, data observability, and data governance practices required for enterprise AI initiatives. Evaluate and recommend emerging AI technologies, frameworks, and best practices that improve data engineering capabilities and operational efficiency. Support Agile delivery by helping manage sprint priorities, backlog refinement, story estimation, daily execution, dependency tracking, and delivery commitments. Coordinate work across onsite, offsite, offshore, and cross-team members to align scope, resolve blockers, and maintain delivery momentum. Create and maintain technical design documentation, data flow diagrams, AI solution architecture diagrams, operational runbooks, and implementation plans for assigned solutions. Drive data quality, performance tuning, monitoring, troubleshooting, and production support for critical data pipelines, AI data services, and Snowflake workloads. Lead technical discussions, architecture reviews, code reviews, root-cause analysis, and problem-solving sessions with engineering, AI, and business partners. Mentor team members on modern data engineering practices, AI-assisted development methodologies, and emerging AI technologies. Promote responsible AI practices, including data privacy, security, governance, compliance, and ethical use of AI solutions.

Required Skills, Knowledge and Experience: Bachelor's degree in Computer Science, Information Technology, Engineering, Data Analytics, Data Science, Artificial Intelligence, or a related technology field, or equivalent job-related experience. Strong hands-on experience with Snowflake data warehouse, including SQL development, performance tuning, data modeling, stored procedures, and scalable data processing patterns. Hands-on experience designing and supporting ETL/ELT pipelines using Nexla and related modern data integration tools. Experience with Apache Airflow or similar job scheduling and workflow orchestration tools, including DAG development, scheduling, monitoring, and troubleshooting. Strong proficiency in SQL and Python for enterprise-scale data engineering solutions. Practical experience using AI-powered development platforms such as Microsoft Copilot, Claude, Cursor, GitHub Copilot, or similar tools to improve engineering productivity and delivery outcomes. Understanding of modern AI and Generative AI concepts, including Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), vector embeddings, prompt engineering, and Agentic AI patterns. Experience designing data architectures and pipelines that support AI, machine learning, analytics, and intelligent automation use cases. Knowledge of data governance, security, privacy, lineage, and compliance requirements related to AI and enterprise data platforms. Ability to lead sprint execution, manage competing priorities, clarify scope, identify risks, and coordinate delivery across multiple technical and business stakeholders. Strong collaboration skills with Product Engineering, Data Engineering, AI/ML Engineering, BI/Analytics, QA, Architecture, and business-facing teams. Experience working with onsite, offsite, offshore, and distributed team members in an Agile delivery environment. Strong analytical, problem-solving, communication, and troubleshooting skills. Self-starter with strong ownership, accountability, attention to detail, and ability to operate independently in a fast-paced environment. Demonstrated ability to learn and adopt emerging AI technologies and frameworks in a rapidly evolving technology landscape.

Preferred but Not Required: Experience with Snowflake on Azure and related Azure data services. Experience with Azure OpenAI, Microsoft Fabric, Databricks, Amazon Bedrock, Anthropic Claude, OpenAI, or similar AI platforms. Experience implementing Retrieval-Augmented Generation (RAG), semantic search, vector databases, AI agents, or Agentic AI frameworks such as LangChain, LangGraph, CrewAI, AutoGen, Semantic Kernel, or similar technologies. Experience building AI-enabled data products, intelligent document processing solutions, conversational AI, or enterprise knowledge retrieval systems. Experience with additional ETL/ELT tools such as Azure Data Factory, Talend, Python-based frameworks, or similar technologies. Experience with DevOps, CI/CD, Infrastructure-as-Code, and MLOps practices supporting AI and data platform deployments. Experience with Jira, Azure DevOps, GitHub, or similar Agile delivery and project tracking tools. Experience supporting enterprise data platforms, data quality frameworks, data governance, metadata management, or Master Data Management initiatives. Experience in the insurance, healthcare, or financial services industry. Relevant certifications in Snowflake, Azure, Generative AI, Claude, Microsoft Copilot, AI Engineering, or Cloud Data Platforms.

For applications and inquiries, contact:View email address on us.fitly.work