Staff AI Test and Evaluation

Responsibilities

• Develop comprehensive test scenarios and simulation environments for assessing agentic AI performance in classified simulations.
• Integrate and validate agentic AI systems on classified platforms and edge hardware.
• Build reusable evaluation pipelines, automated test harnesses, and monitoring dashboards for continuous validation.
• Define metrics and systematic historical performance capture for AI and traditional models.
• Partner with cross-functional teams to define requirements, document test results, and contribute to model cards and deployment readiness reviews.
• Architect, build, and maintain end-to-end evaluation and validation infrastructure for agentic AI and ML models.
• Analyze and resolve issues found during evaluation and deployment to ensure reliability across releases.

Requirements

• Bachelor's or master's degree in computer science, machine learning, software engineering, computer engineering, electrical engineering, or a related technical field.
• At least 12+ years of hands-on experience writing production-grade code, with strong Python skills.
• Experience owning evaluation or automated testing for production ML or software systems, including scenarios, metrics, regression suites, and CI infrastructure.
• Hands-on experience designing evaluation methodologies, metrics, benchmarks, and real-world assessments for AI or machine learning models.
• Experience designing or working extensively with simulation environments.
• Working knowledge of defining, capturing, and analyzing performance metrics over time.
• Ability to contribute to complex systems and established codebases, coordinate across teams, and document results.
• Eligibility for a US security clearance.
• Preferred experience includes ML test and evaluation, SDET for ML systems, simulation engineering, AI/ML technical program management, agentic AI evaluation, classified or edge deployment, model cards, readiness reviews, aerospace or defense test and evaluation, MLOps, monitoring dashboards, and Go or C++.

Benefits

• Full-time employees receive comprehensive competitive benefits available at little to no cost.
• The role offers a stated US salary range of $253,000—$336,000 in base salary, with equity grants included in the majority of full-time offers.