AI Agent Evaluation Engineer - Remote

YO AI Labs

Austin, TX

Remote job

Job Description

Senior Software Engineer

Job Type: Contractor (~15 hours/week)
Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization . No prior AI experience is required.

Key Responsibilities

Create reinforcement learning environments for software engineering tasks.
Design tasks involving bug fixing, feature development, refactoring, and performance optimization .
Build deterministic verification systems and golden reference solutions.
Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
Develop realistic, reproducible software engineering scenarios.
Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
Document solutions and provide clear technical feedback.

Required Skills

Strong proficiency in Python 3, Java, Rust, C++, or TypeScript .
Strong understanding of algorithms and data structures .
Experience with bug fixing and debugging complex software issues .
Proven experience in feature implementation and codebase refactoring .
Strong knowledge of performance optimization and tuning .
Excellent written and verbal communication.
Strong attention to detail.

Preferred Qualifications

Experience working with large-scale or distributed codebases .
Familiarity with AI/ML systems is a plus but not required.
Experience with rigorous code reviews and software engineering best practices .
Experience working effectively in remote or cross-functional teams.

Hiring Process

Submit an application and screening questions.
Complete an AI interview (~30 minutes).
Complete a technical assessment, if required.
Hiring Manager review.

Compensation

Compensation is output-based , with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding .

Apply

Vacancy posted 8 days ago

Similar jobs that could be interesting for youBased on the AI Agent Evaluation Engineer - Remote in Austin, TX vacancy

Remote: AI Agent Evaluation Scenario Designer
...technology company is looking for a detail-oriented individual to design structured evaluation scenarios for AI agents. This entry-level part-time role allows you to contribute remotely, creating test cases for LLM-based agents. The ideal candidate will have a background...
Remote work
Part time
Mind Rift
United States
1 day ago
Remote QA Analyst for Autonomous AI Agent Evaluation
$60 per hour
...A leading AI firm in Austin is looking for QA experts to validate... ...and improve AI systems. This remote, freelance role requires... ...detail. Candidates will review AI evaluation tasks, identify inconsistencies... ...define expected behaviors for agents. Ideal applicants have experience...
Remote work
Freelance
Mind Rift
United States
1 day ago
Staff Software Engineer, Systems Infrastructure - Agent Evaluation
$175k - $287k
...the business needs of the team. LinkedIn’s Core AI is building the Evaluation Operating System (EOS), a foundational Agent Evaluation platform that defines how all AI... ...infrastructure for all LinkedIn AI Agents.As a Staff Engineer, you will own the end-to-end technical vision,...
Suggested
For contractors
Work experience placement
Work at office
Flexible hours
Linkedin
Mountain View, CA
1 day ago
Senior Software Engineer - AI Evaluation / Coding Agents
...Freelance · Remote · North America, LATAM, or India About Turing... ..., working with frontier AI labs to accelerate model development... ...high-quality training data, evaluations, and engineering talent. About the Role... ..., you’ll work with coding agents across real-world...
Remote work
Hourly pay
Temporary work
For contractors
Freelance
Immediate start
Turing
Remote
24 days ago
RSI Evaluation & Agent Training Data Engineer
...TELUS Digital is sourcing elite AI Evaluation Experts and Benchmark Engineers to support next-generation Recursive... ...Authoring self-contained, hermetic agent task packages (Docker containers, ground... ...(AI Community) Location: 100% Remote (United States, Europe, India, and...
Remote work
Freelance
Local area
TELUS Digital AI Data Solutions
United States
4 days ago
AI Agent & Automation Engineer
...Tyndale is expanding our AI and automation capabilities... ...seeking an experienced AI Agent & Automation Engineer to build production solutions... ...support. HYBRID/REMOTE: Tyndale supports a strong... ...platforms, AI agent testing/evaluation, containerization and operational...
Remote work
Casual work
1 day per week
Tyndale
Pipersville, PA
a month ago
Software Engineers: Paid Code Review for AI Agent Evaluation
$135.2k
...to benchmark artificial intelligence agents. Creating robust evaluation harnesses ensures that AI models are tested against realistic software engineering scenarios. This work directly feeds... ...objectives. How It Works During this remote session, you will review a series of...
Remote work
Hourly pay
Contract work
Terac
Remote
3 days ago
Mechanical Engineer for AI Training and Evaluation
$70 - $125 per hour
...Apply hands-on mechanical engineering judgment to help evaluate and improve advanced AI systems. Your design and analysis standards will serve as the benchmark for... ...raising unclear instructions. Work Terms Remote, hourly independent-contractor work. Project invitations...
Remote work
Hourly pay
For contractors
SaidGig
United States
19 days ago
Electrical Engineer for AI Training and Evaluation
$70 - $120 per hour
...Role Overview Apply your electrical engineering judgment to help evaluate and improve frontier AI systems. Your expertise will inform the standards used to assess... ...with ambiguous assignments. Work Terms Remote, hourly independent contractor work. Project invitations...
Remote work
Hourly pay
For contractors
SaidGig
United States
19 days ago
Mechanical Engineer for AI Model Evaluation
$60 - $90 per hour
...Overview Apply hands-on mechanical engineering judgment to improve how advanced AI models reason through real-world... ...solutions, and create rigorous evaluations grounded in industry practice.... ...structured feedback. Work Terms Remote role open to candidates located...
Remote work
Hourly pay
Full time
SaidGig
United States
17 days ago
AI Agent and Automation Specialist
...Overview DecisionPoint seeks an AI Agent and Automation Specialist to design, build... ...support. This position is fully remote. Duties & Responsibilities... ...and triage logic using AI models, rules engines, and workflow platforms. ~ Integrate...
Remote work
For contractors
Local area
DecisionPoint | Cortek
United States
19 hours ago
Senior Structural Engineer (OpenSees | AI Training & Evaluation)
$80 - $110 per hour
...enterprise client is seeking experienced Structural Engineers to help train and evaluate next-generation AI systems. This opportunity is part of a large-scale... ...engineering. Work independently in a fully remote, flexible environment. Qualifications Requirements...
Remote work
Hourly pay
Contract work
10 hours per week
Flexible hours
Lifted, an Upwork Company™
Chicago, IL
14 days ago
STEM Professional for AI Coding Agent Evaluation
$80 - $100 per hour
...and improve next-generation AI coding agents. In this contractor role you... ...are used for complex software engineering work, shaping model behavior... ...development. Critically evaluate AI agents in diverse... ...type: Contractor. Location: Remote. Work will involve contributing...
Remote work
Hourly pay
For contractors
SaidGig
United States
6 days ago
Cloud Engineer for AI Evaluation & Automation (Remote)
...is a research lab that builds the evaluation and training data frontier AI labs use to measure what models can... ...real professional work. We hire cloud engineers as contractors to author and grade... ...run on. This role is fully remote and async, with no fixed hours. It...
Remote job
Weekly pay
For contractors
10 hours per week
Jobleads-US
Sunnyvale, CA
2 days ago
Senior Frontier Agents Engineer (Applied AI)
$216k - $270k
About Scale AIScale AI is the data foundation for AI, helping... ...telecommunications to build production AI agents that automate complex... ...remains one of the hardest engineering challenges.As a Senior... ...enterprise customers to design, evaluate, and deploy intelligent systems...
Full time
Scale AI
New York, NY
4 days ago
IT Helpdesk Agent Evaluator
...IT Helpdesk Agent Evaluator is a remote evaluation track for reviewing it helpdesk agent evaluation prompts and responses against AuraOne's quality... ...team can use to retrain. Why this role matters AI data reviewers help turn it helpdesk agent evaluation outputs...
Remote job
Hourly pay
For contractors
10 hours per week
AuraOne Human Data
Remote
more than 2 months ago
Investor Materials Expert - Evaluator - AI Trainer
$80 - $120 per hour
...elite creative and technical talent with leading AI research labs. Headquartered in San Francisco... ...Investor materials / fundraising / pitchbook Evaluator Type: Contract Compensation: $80–$120/hour Location: Remote Role Responsibilities Evaluate AI-...
Remote work
Contract work
Summer work
Work at office
Mercor
Charlotte, NC
5 days ago
Engineer for AI Training and Evaluation
$60 - $125 per hour
...Role Overview Apply engineering judgment to help evaluate and improve advanced AI systems. Your professional standards and technical expertise will inform the criteria... ..., and materials engineering. Work Terms Remote, hourly work. This is an ongoing talent pool...
Remote work
Hourly pay
SaidGig
United States
1 day ago
Agent Engineer
$140k - $164.7k
...Role Summary You will own the AI agent development lifecycle at ScottsMiracle... ..., voice, email). Implement prompt engineering, guardrails, and evaluation pipelines to ensure agents meet quality... ...corporate bonus programs. For remote roles where the final candidate...
Remote work
Jobleads-US
Kentucky
1 day ago
Senior Software Engineer, AI Evaluation and Coding Agents
$100 - $150 per hour
...Role Overview Help improve AI coding agents by reviewing their work across... ...will help create the evaluation signals used to strengthen model... ...closely with researchers and engineers to convert qualitative technical... ...required. Work Terms Remote independent-contractor...
Remote work
Hourly pay
For contractors
Immediate start
SaidGig
United States
19 hours ago
Senior Forward Deployed Engineer - Okta for AI Agents
$200k - $275k
...Secure Every Identity, from AI to Human Identity is the key... ...let's talk. About Okta for AI Agents Okta secures access for 20,00... ...identity, platform, and security engineering teams, write production code... ...outcomes end to end. #LI-Remote P25257_3456702 Below is the...
Remote work
Local area
Worldwide
Flexible hours
Shift work
Okta
Bellevue, WA
more than 2 months ago
DevOps / SRE / Cloud Engineer (Coding Agent Experience)
$85 per hour
...DevOps / SRE / Cloud Engineer (Coding Agent Experience) is a remote engineering review track for evaluating production code, debugging traces, and developer-facing AI outputs against real-world correctness standards. Reviewers reproduce failures, write the unit test the...
Remote job
For contractors
10 hours per week
AuraOne Human Data
Remote
12 days ago
Agent Operations Engineer - Fully Remote | Upto $500/task Task based
$500 per month
...technical talent with leading AI research labs. Headquartered... ...Jack Dorsey . Position: Agent Engineer Type: Contract Compensation... ...500 per task Location: Remote Role Responsibilities... ...Ensure agent reliability and evaluate performance improvements....
Remote work
Contract work
Summer work
Mercor
New York, NY
5 days ago
AI Agent Evaluation Engineer - Remote
...Description Senior Software Engineer Job Type: Contractor... .../week) Location: Remote Job Summary We are... ...to support an AI training project by creating... ...learning environments that evaluate AI models on complex... ...solutions. Evaluate AI agents' ability to reason...
Remote job
For contractors
YO AI Labs
Dallas, TX
8 days ago
Senior AI Architect for Enterprise Agent Platform (Remote)
...BairesDev is seeking a Principal AI Engineer to set the technical direction and own the architecture of a major enterprise AI assistant... ...You will shape the system from the ground up, ensure robust evaluation, and drive best practices in CI/CD, observability, and scalable...
Remote job
Jobleads-US
Palo Alto, CA
5 days ago
Engineer II, Gen AI
$94.58k - $110k
...opportunity to join our team as an Engineer II.In this role, the AI Engineer will design, implement, and... ...solutions that support NYU Langone Healths Remote Patient Monitoring (RPM) initiatives... ..., validation, training, evaluation, and model deployment across the RPM...
Remote work
Full time
NYU Langone Medical Center
New York, NY
4 days ago
Enterprise Agent Systems Engineer
...seeking a highly technical Enterprise Agent Systems Engineer to build and deploy advanced agent... ...strong software engineering with applied AI research judgment. You will work... ...pipelines, post-training workflows, and evaluation infrastructure required for production...
MaxIT Consulting - Max Corporate Group
San Francisco, CA
3 days ago
Senior Software and System Development Engineer I
$84.5k - $105.6k
...~ As a digital tools engineer, you will develop, deploy... ...tools, knowledge retrieval agents Stakeholder... ...engineering processes, evaluate emerging technologies[NM... ...SharePoint APIs o Applying AI/LLM solutions o... ...is an onsite job. One remote workday per week may be...
Remote work
Full time
Temporary work
Work experience placement
Work at office
Relocation package
Honda Dev. and Mfg. of Am.,LLC
Raymond, OH
7 days ago
Senior Software Engineer - Agent Systems
...individual investors and institutional clients. The Agent Systems team is a 0→1 engineering group building internal AI-powered agents that interact directly with... ...Familiarity with prompt engineering, evaluation frameworks, or model performance tuning Experience...
Full time
Local area
Kraken
United Kingdom
5 days ago
Principal Product Manager - Workday AI - Agent Factory
$196k - $294k
...company and a leading AI platform for managing people, money, and agents, we’re shaping the future... ...agent architectures, evaluation frameworks, and customer... ...viability before committing engineering resourcesDesign... ...worlds: in-person time and remote. Our approach enables...
Remote work
Full time
Work at office
Home office
Flexible hours
Weekend work
Workday
Pleasanton, CA
2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Evaluation Engineer - Remote. Be the first to apply!