AI Data Engineer

Data Engineer

Must Have Technical/Functional Skills

• Strong hands-on experience in Python, SQL and data engineering.
• Experience designing and developing ETL/ELT pipelines for large-scale data processing.
• Strong knowledge of PySpark / Spark for data transformation and processing.
• Experience with AWS data services such as Glue, S3, Redshift and related cloud services.
• Hands-on experience with Apache Airflow or equivalent orchestration tools.
• Strong understanding of data transformation, cleansing, validation, reconciliation and data-quality frameworks.
• Experience working with structured, semi-structured and unstructured data.
• Knowledge of batch and streaming data pipelines, preferably Kafka.
• Experience building data pipelines supporting AI/ML, GenAI, LLM and RAG use cases.
• Experience with CI/CD, Git, Docker and Kubernetes/EKS for production deployments.
• Good understanding of data modeling, performance optimization and production support.

Roles & Responsibilities

• Design, develop and maintain scalable end-to-end ETL/ELT and data pipelines.
• Build data processing and transformation solutions using Python, SQL and PySpark.
• Develop batch and streaming pipelines to integrate data from databases, APIs, files and enterprise source systems.
• Implement data-quality checks, validation, reconciliation, deduplication and exception handling.
• Optimize data pipelines, SQL queries and Spark workloads for high-volume processing.
• Prepare and transform datasets required for AI/ML and Generative AI applications.
• Develop data ingestion and preparation pipelines supporting LLM/RAG solutions and vector databases.
• Collaborate with AI/ML engineers, application teams and business stakeholders on downstream data requirements.
• Implement automated deployment and monitoring through CI/CD and DevOps practices.
• Support production deployments, monitoring, troubleshooting and root-cause analysis for data-related issues.

Salary Range: $90,000 to $115,000 per year