AI/ML & GenAI Observability Engineer

. AI/ML & GenAI Observability


Job Responsibilities:
• Design, develop, and operationalize end-to-end data pipelines using distributed processing frameworks (e.g., Spark, Kafka, dbt) to handle high-volume ingestion, transformation, and serving of structured and unstructured datasets across cloud-based infrastructure
• Author, deploy, and maintain complex Apache Airflow DAGs, including dynamic task generation, cross-DAG dependencies, custom operators/sensors, SLA monitoring, and backfill strategies in a production-grade orchestration environment
• Architect and implement GenAI observability pipelines that capture LLM inference telemetry, including token usage, latency distributions, prompt/response logging, and model versioning metadata at scale
• Build automated LLM evaluation frameworks leveraging scoring metrics such as ROUGE, BERTScore, G-Eval, or custom rubric-based evaluators to systematically assess generative model output quality, factual grounding, and safety compliance
• Develop and maintain metrics computation layers using tools such as Great Expectations, Evidently AI, or custom Python-based frameworks to monitor data drift, model degradation, and evaluation score trends over time
• Integrate observability outputs into centralized ML monitoring platforms (e.g., MLflow, Weights & Biases, LangSmith, or internal tooling) to enable real-time and historical analysis of generative AI system performance
• Enforce data quality and pipeline reliability through schema validation, idempotency patterns, dead-letter queue handling, and automated alerting using tools such as PagerDuty or Grafana
• Optimize pipeline performance and cost efficiency by profiling query execution plans, implementing partitioning/bucketing strategies, and leveraging cloud-native services (e.g.,, BigQuery, Redshift, ) for scalable compute