Senior Software Engineer in Test

Responsibilities

• Build and maintain a versioned golden-set library of grounding data and ground-truth results with senior claims examiners.
• Design automated model-as-a-judge workflows and custom rubrics for factual grounding and policy compliance.
• Develop semantic assertion frameworks that validate semantic equivalence and numerical accuracy in agent outputs.
• Audit Vertex AI traces to verify that mandatory MCP tools are invoked with correct arguments.
• Validate orchestration priorities across data and knowledge, orchestration, agentic reasoning, and tooling layers.
• Ensure autonomous decisions are traceable to specific SOP sentences and live API data points.
• Integrate automated regression runs with Vertex AI Prompt Management for every prompt or model update.
• Own automated side-by-side comparisons between new and production agent versions to detect logic drift.
• Build Vertex AI and ADK mocking infrastructure for high-volume, low-cost logic testing.

Requirements

• 5+ years of experience in software testing and development.
• Expert Python and pytest skills, including custom mocking frameworks for external APIs such as Vertex AI and ADK.
• Hands-on experience with Vertex AI Experiments, Auto-SxS, and Cloud Logging for trace analysis.
• Expert-level SQL with BigQuery and Pandas skills for comparing large datasets and identifying adjudication discrepancies.
• Ability to analyze system instructions and refine prompts based on failed test cases.
• Experience testing multi-layer systems involving RAG with Vertex AI Search, state management with LangGraph, and function calling.
• Familiarity with healthcare claims adjudication concepts such as pend reason codes, COB, eligibility, and stop-loss is preferred.
• Understanding of HIPAA/PHI handling and preparation of test evidence for DOL and DOI regulators is preferred.
• Experience with shadow-mode monitoring and comparing agent decisions against human expert baselines is preferred.

Benefits

• Hybrid work based in Lehi, Utah or Plano, Texas, with at least two weekdays in the office.
• Health insurance, 401(k), and paid time off.
• Professional development through internal mobility, mentorship programs, and tailored courses.
• Flexible work arrangements and work-life balance support.