Senior Python Engineer - Data Pipelines

This is a hybrid role based in Hyderabad, India.

Role Summary

Seeking a hands-on senior Python engineer to own data pipelines end to end (ingestion, transformation, orchestration, quality, and delivery) at enterprise scale. Pipelines feed analytics, search, and GenAI/Agentic AI platforms.

Responsibilities

- Design, build, and own production data pipelines in Python (batch and near-real-time).

- Operate orchestrated workflows (AWS Step Functions, Airflow, Dagster, or Prefect) with retries, idempotency, and backfills.

- Model data at scale: partitioning, Parquet/Avro/Iceberg, schema evolution, CDC.

- Establish data quality and observability (Great Expectations/Pandera, SLAs, lineage, alerting).

- Build document-processing and RAG retrieval pipelines (chunking, embeddings, vector and hybrid search).

- Build serving APIs with FastAPI, Flask, or Django REST.

- Set Python engineering standards; mentor junior engineers; liaise with business stakeholders.

Qualifications

- 8–12 years of software engineering, 5–6+ years building production data pipelines in Python.

- Expert modern Python (3.10+), typing, asyncio/concurrency, pandas/Polars, PyArrow, PySpark/Dask/Ray.

- Strong SQL and data modeling; Snowflake, Redshift, Iceberg/Delta, dbt.

- Strong AWS: Lambda, Step Functions, S3, Glue, EMR, IAM, CloudWatch; IaC (CDK/Terraform).

- pytest, packaging (uv/Poetry), CI/CD.

- Strongly preferred: RAG, AWS Bedrock, LangChain/LlamaIndex, agentic frameworks, MCP servers, GenAI evaluation.

- Nice to have: Kafka/Kinesis/Flink streaming, C#/.NET, regulated-industry experience.