QA/Test Engineer for AI Model Evaluation

Validate the integrity of complex, multi-step evaluation tasks used to benchmark frontier generative AI models. You will design and execute rigorous test cases, probe edge cases, and debug task environments so each task is unambiguous, correctly graded, and robust to shortcuts. Individual tasks typically represent one to two days of expert effort and span multiple technical skills. You will operate in a close feedback loop with the lab''s researchers and task authors to ensure benchmark results remain trustworthy.

• Design checks and test cases that confirm each task behaves as intended, including difficult edge cases.
• Review tasks and reference solutions in detail before finalization, identifying ambiguity, grading gaps, and missing assumptions.
• Debug task logic and verification code, actively using Python to investigate and fix failures.
• Help create simple, repeatable quality checklists and provide actionable feedback that task authors can apply quickly.
• Monitor AI agent runs for shortcuts or grading weaknesses to protect benchmark validity.
• Collaborate closely with researchers and task authors in an iterative feedback loop to improve task quality and evaluation processes.

• MSc or PhD in a STEM field, or equivalent practical experience in a research-heavy or engineering-heavy domain.
• At least 1 year of experience in test engineering, quality assurance, or a research or software engineering role that included strong quality ownership.
• Proven ability to design test cases and quality-review processes, and to debug complex systems end-to-end.
• Working proficiency in Python and Git, with comfort navigating unfamiliar codebases and runtime environments.
• Exceptional attention to detail and clear written documentation habits.
• Prior experience with AI training, model evaluation, or quality review of AI-generated outputs is preferred.
• A perfectionist mindset, creativity in finding what others missed, and the ability to work independently on ambiguous, open-ended problems.
• Availability to engage reliably for approximately 35 hours per week.

• Status: Full-time W-2 employment with Cincinnatus LLC, acting as the employer of record for this placement.
• Placement: Opportunity to be placed at a leading AI lab as part of their extended workforce, working within the client team and enterprise workflows.
• Location: Fully remote within the United States, candidates must be able to work from the U.S.
• Schedule: Approximately 35 hours per week.
• Engagement type: Role-based employment, not a freelance or project-by-project arrangement; integration with client teams and standard enterprise processes is expected.

• Pay rate: $60.00 to $90.00 per hour, paid on an hourly basis.

• Employment, onboarding, payroll, and benefits for this role are administered by Cincinnatus LLC, the employer of record.
• Opportunities may be discovered through third-party talent platforms, but final hiring and employment administration are handled by Cincinnatus LLC.
• Cincinnatus LLC is an Equal Employment Opportunity employer and provides reasonable accommodations for qualified individuals with disabilities throughout the application process.