AI Agent Evaluation Engineer - Remote
YO AI Labs
Los Angeles, CA
- Remote job
Job Description
Job Description
Senior Software Engineer
Job Type: Contractor (~15 hours/week)
Location: Remote
We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.
You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization . No prior AI experience is required.
Key Responsibilities- Create reinforcement learning environments for software engineering tasks.
- Design tasks involving bug fixing, feature development, refactoring, and performance optimization .
- Build deterministic verification systems and golden reference solutions.
- Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
- Develop realistic, reproducible software engineering scenarios.
- Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
- Document solutions and provide clear technical feedback.
- Strong proficiency in Python 3, Java, Rust, C++, or TypeScript .
- Strong understanding of algorithms and data structures .
- Experience with bug fixing and debugging complex software issues .
- Proven experience in feature implementation and codebase refactoring .
- Strong knowledge of performance optimization and tuning .
- Excellent written and verbal communication.
- Strong attention to detail.
- Experience working with large-scale or distributed codebases .
- Familiarity with AI/ML systems is a plus but not required.
- Experience with rigorous code reviews and software engineering best practices .
- Experience working effectively in remote or cross-functional teams.
- Submit an application and screening questions.
- Complete an AI interview (~30 minutes).
- Complete a technical assessment, if required.
- Hiring Manager review.
Compensation is output-based , with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.
AvailabilitySelected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding .
Vacancy posted 6 days ago
Similar jobs that could be interesting for youBased on the AI Agent Evaluation Engineer - Remote in Los Angeles, CA vacancy
- ...technology company is looking for a detail-oriented individual to design structured evaluation scenarios for AI agents. This entry-level part-time role allows you to contribute remotely, creating test cases for LLM-based agents. The ideal candidate will have a background...Remote workPart time
Mind Rift
United States4 days ago $80 per hour
A leading AI consultancy is seeking a detail-oriented individual to design evaluation scenarios for LLM-based agents. This part-time, remote role allows you to create structured test cases while working flexibly around your commitments. The ideal candidate holds a relevant...Remote workPart time$80 per hour
...-time opportunity focused on quality assurance for autonomous AI agents. You will analyze complex systems, review tasks for logic, and... ...analytical and detail-oriented skills, with experience in policy evaluation or logic puzzles preferred. Compensation can reach up to $80/...Remote jobPart timeFlexible hours$80 per hour
A technology firm is seeking QAs for autonomous AI agents to validate and improve task structures within a new project. Candidates will... ...thinkers with strong attention to detail and experience in evaluating scenarios. This flexible, project-based role offers competitive...Remote jobFlexible hours$60 per hour
...contributors for a part-time QA project focused on autonomous AI agents. This flexible remote opportunity requires strong analytical and critical... ...with structured data formats. Candidates will review evaluation tasks, identify inconsistencies, and help define expected...Remote jobPart timeFlexible hours$60 per hour
...A leading AI firm in Austin is looking for QA experts to validate... ...and improve AI systems. This remote, freelance role requires... ...detail. Candidates will review AI evaluation tasks, identify inconsistencies... ...define expected behaviors for agents. Ideal applicants have experience...Remote workFreelanceMind Rift
United States4 days ago$250k - $300k
Senior Software Engineer — Infra Agent Systems San Francisco About the Role Together AI runs one of the largest GPU fleets in the world... ...and retrieval, orchestration, evaluation, and the tooling that enables... ...as flexibility in terms of remote work. The US base salary range...Remote workFull time$60 per hour
...ethically shape the future of AI. What We Do The... ...thrive in ambiguity, enjoy remote asynchronous work, and... ...systems are tested and evaluated, we want to hear from... ...experts for autonomous AI agents in a project focused on... ...to LLMs, prompt engineering, or AI‑generated content...Remote workFreelanceFlexible hours- ...leading security-first enterprise AI company, is seeking a Member... ...Staff in the Safety for Agents team to advance safer LLMs through... ...post-training algorithms, and evaluation methods. You'll collaborate with... ...across global offices, while remote-friendly policies support...Remote workFlexible hours
$175k - $287k
...the business needs of the team. LinkedIn’s Core AI is building the Evaluation Operating System (EOS), a foundational Agent Evaluation platform that defines how all AI... ...infrastructure for all LinkedIn AI Agents.As a Staff Engineer, you will own the end-to-end technical vision,...For contractorsWork experience placementWork at officeFlexible hoursLinkedin
Mountain View, CA4 days ago- ...Freelance · Remote · North America, LATAM, or India About Turing... ..., working with frontier AI labs to accelerate model development... ...high-quality training data, evaluations, and engineering talent. About the Role... ..., you’ll work with coding agents across real-world...Remote workHourly payTemporary workFor contractorsFreelanceImmediate start
Turing
Remote22 days ago - ...Tyndale is expanding our AI and automation capabilities... ...seeking an experienced AI Agent & Automation Engineer to build production solutions... ...support. HYBRID/REMOTE: Tyndale supports a strong... ...platforms, AI agent testing/evaluation, containerization and operational...Remote workCasual work1 day per week
Tyndale
Pipersville, PAa month ago - ...Description Senior Software Engineer Job Type: Contractor... .../week) Location: Remote Job Summary We are... ...to support an AI training project by creating... ...learning environments that evaluate AI models on complex... ...solutions. Evaluate AI agents' ability to reason...Remote workFor contractors
YO AI Labs
Boston, MAa month ago - YO AI Labs seeks an experienced Civil Engineer to contribute to AI evaluation projects by creating realistic civil tasks, applying structural design and regulatory codes, and developing precise solutions. As a contractor, you will produce engineering drawings, reports,...Remote jobFor contractors
$135.2k
...to benchmark artificial intelligence agents. Creating robust evaluation harnesses ensures that AI models are tested against realistic software engineering scenarios. This work directly feeds... ...objectives. How It Works During this remote session, you will review a series of...Remote workHourly payContract work$70 - $125 per hour
...Apply hands-on mechanical engineering judgment to help evaluate and improve advanced AI systems. Your design and analysis standards will serve as the benchmark for... ...raising unclear instructions. Work Terms Remote, hourly independent-contractor work. Project invitations...Remote workHourly payFor contractorsSaidGig
United States17 days ago$70 - $120 per hour
...Role Overview Apply your electrical engineering judgment to help evaluate and improve frontier AI systems. Your expertise will inform the standards used to assess... ...with ambiguous assignments. Work Terms Remote, hourly independent contractor work. Project invitations...Remote workHourly payFor contractorsSaidGig
United States17 days ago$60 - $90 per hour
...Overview Apply hands-on mechanical engineering judgment to improve how advanced AI models reason through real-world... ...solutions, and create rigorous evaluations grounded in industry practice.... ...structured feedback. Work Terms Remote role open to candidates located...Remote workHourly payFull timeSaidGig
United States15 days ago- ...Overview DecisionPoint seeks an AI Agent and Automation Specialist to design, build... ...support. This position is fully remote. Duties & Responsibilities... ...and triage logic using AI models, rules engines, and workflow platforms. ~ Integrate...Remote workFor contractorsLocal area
DecisionPoint | Cortek
United States3 days ago - ...IDR is seeking a Software Engineer - AI Evaluation & Automation to join one of our top clients for a remote opportunity. This role is within the technology sector, focusing... ...workflows. Some familiarity with AI, LLM, or agent evaluation processes, including understanding...Remote workTemporary work
IDR
Santa Clara, CA21 days ago - ...is a research lab that builds the evaluation and training data frontier AI labs use to measure what models can... ...real professional work. We hire cloud engineers as contractors to author and grade... ...run on. This role is fully remote and async, with no fixed hours. It...Remote jobWeekly payFor contractors10 hours per week
$80 - $110 per hour
...enterprise client is seeking experienced Structural Engineers to help train and evaluate next-generation AI systems. This opportunity is part of a large-scale... ...engineering. Work independently in a fully remote, flexible environment. Qualifications Requirements...Remote workHourly payContract work10 hours per weekFlexible hoursLifted, an Upwork Company™
Chicago, IL12 days ago$80 - $100 per hour
...and improve next-generation AI coding agents. In this contractor role you... ...are used for complex software engineering work, shaping model behavior... ...development. Critically evaluate AI agents in diverse... ...type: Contractor. Location: Remote. Work will involve contributing...Remote workHourly payFor contractorsSaidGig
United States4 days ago$216k - $270k
About Scale AIScale AI is the data foundation for AI, helping... ...telecommunications to build production AI agents that automate complex... ...remains one of the hardest engineering challenges.As a Senior... ...enterprise customers to design, evaluate, and deploy intelligent systems...Full timeScale AI
New York, NY2 days ago- ...AI Agent Interaction Specialist - Dialogue & Conversation Evaluation is a remote evaluation track for reviewing ai agent interaction evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind...Remote jobHourly payFor contractors10 hours per week
$126.6k - $194.7k
...Director, Enterprise Quality Engineering and Agentic Evaluation to build and lead a modern... ...programs, and AI-enabled solutions. The successful... ...integrations, data products, and AI agents. ~ Lead a centralized... ...or in the future. #LI-Remote The base pay range shown...Remote workFull timeWork experience placementLocal area$60 - $125 per hour
...Role Overview Apply engineering judgment to help evaluate and improve advanced AI systems. Your professional standards and technical expertise will inform the criteria... ..., and materials engineering. Work Terms Remote, hourly work. This is an ongoing talent pool...Remote workHourly paySaidGig
United States3 days ago$200k - $275k
...Secure Every Identity, from AI to Human Identity is the key... ...let's talk. About Okta for AI Agents Okta secures access for 20,00... ...identity, platform, and security engineering teams, write production code... ...outcomes end to end. #LI-Remote P25257_3456702 Below is the...Remote workLocal areaWorldwideFlexible hoursShift work$85 per hour
...DevOps / SRE / Cloud Engineer (Coding Agent Experience) is a remote engineering review track for evaluating production code, debugging traces, and developer-facing AI outputs against real-world correctness standards. Reviewers reproduce failures, write the unit test the...Remote jobFor contractors10 hours per week$500 per month
...technical talent with leading AI research labs. Headquartered... ...Jack Dorsey . Position: Agent Engineer Type: Contract Compensation... ...500 per task Location: Remote Role Responsibilities... ...Ensure agent reliability and evaluate performance improvements....Remote workContract workSummer workMercor
New York, NY3 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Evaluation Engineer - Remote. Be the first to apply!
Related searches
- operations agent Los Angeles, CA
- work from home chat agent Los Angeles, CA
- executive protection agent Los Angeles, CA
- freight agent no experience Los Angeles, CA
- import export agent Los Angeles, CA
- signing agent Los Angeles, CA
- agent assistant Los Angeles, CA
- tsa agent Los Angeles, CA
- telemarketer - state farm agent team member Los Angeles, CA
- agent Los Angeles, CA