Senior ETL Tester with AI Testing Experience

Job Description

Job Summary

We are seeking an experienced Senior QA / ETL Test Lead with strong expertise in ETL testing, SQL, Azure Databricks, Python automation, and Agentic AI evaluation. The ideal candidate will lead testing strategies, validate data integrity across legacy and modern data platforms, and ensure AI-generated queries and transformation logic meet accuracy, security, and compliance standards.

Roles and Responsibilities

\n
• Own the overall ETL test strategy, test planning, reconciliation methodology, and risk-based test coverage.
\n
• Design source-to-target validation across data ingestion, transformation, and refined data layers.
\n
• Build and maintain golden reference datasets to evaluate AI-generated queries and transformation logic.
\n
• Develop automated regression frameworks to compare new pipeline outputs with existing production data.
\n
• Perform row-count, aggregate, field-level, data-type, null, duplicate, and referential integrity validations.
\n
• Evaluate AI agents for accuracy, consistency, hallucinations, incorrect joins, missing fields, and confidence calibration.
\n
• Use MLflow or similar tools to track evaluation results, compare model versions, and establish regression gates.
\n
• Validate healthcare data privacy, PHI protection, multi-tenant isolation, and facility-level data access controls.
\n
• Validate data-sharing contracts and ensure generated queries comply with permitted access restrictions.
\n
• Perform query optimization and acquisition-cost validation against baseline queries.
\n
• Lead and mentor QA engineers, review test coverage, manage defects, and report quality metrics.
\n
• Ensure compatibility across Azure Databricks, SQL Server, Vertica, and other data platforms.
\n
• Prepare test evidence and quality reports for client and executive reviews.
\n

Mandatory Skills

\n
• 10+ years of experience in data / ETL testing, including test strategy ownership for a major migration or re-platforming project.
\n
• Strong SQL skills, including complex queries and data reconciliation.
\n
• Hands-on experience with Azure Databricks, PySpark, and Python automation.
\n
• Experience with data quality and automation frameworks such as Great Expectations, dbt tests, pytest, or equivalent.
\n
• Expertise in source-to-target validation, data integrity, regression testing, and test strategy development.
\n
• Experience testing healthcare data, including PHI-scoped, de-identified, or synthetic datasets.
\n
• Experience with at least two data platforms, such as Databricks, SQL Server, or Vertica.
\n
• Strong leadership, defect management, stakeholder communication, and reporting skills.
\n
• Experience with web testing and test automation tools.
\n

Preferred Skills

\n
• Experience testing Agentic AI, LLM, or GenAI systems.
\n
• Familiarity with MLflow, LangSmith, Weights & Biases, or similar evaluation tools.
\n
• Knowledge of LLM-as-a-judge techniques, statistical sampling, confidence intervals, and model calibration.
\n
• Epic EHR experience, particularly Caboodle or Clarity.
\n
• Familiarity with healthcare coding standards such as ICD, CPT, SNOMED, and LOINC.
\n
• Experience with multi-tenant data isolation and SQL-generation system testing.
\n
• Experience presenting test evidence and quality findings to client executives.
\n
• Knowledge of software design principles, SOLID, and Domain-Driven Design.
\n