AI Agent Evaluation Engineer - Remote
YO AI Labs
San Jose, CA
- Remote job
Job Description
Job Description
Senior Software Engineer
Job Type: Contractor (~15 hours/week)
Location: Remote
We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.
You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization . No prior AI experience is required.
Key Responsibilities- Create reinforcement learning environments for software engineering tasks.
- Design tasks involving bug fixing, feature development, refactoring, and performance optimization .
- Build deterministic verification systems and golden reference solutions.
- Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
- Develop realistic, reproducible software engineering scenarios.
- Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
- Document solutions and provide clear technical feedback.
- Strong proficiency in Python 3, Java, Rust, C++, or TypeScript .
- Strong understanding of algorithms and data structures .
- Experience with bug fixing and debugging complex software issues .
- Proven experience in feature implementation and codebase refactoring .
- Strong knowledge of performance optimization and tuning .
- Excellent written and verbal communication.
- Strong attention to detail.
- Experience working with large-scale or distributed codebases .
- Familiarity with AI/ML systems is a plus but not required.
- Experience with rigorous code reviews and software engineering best practices .
- Experience working effectively in remote or cross-functional teams.
- Submit an application and screening questions.
- Complete an AI interview (~30 minutes).
- Complete a technical assessment, if required.
- Hiring Manager review.
Compensation is output-based , with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.
AvailabilitySelected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding .
Vacancy posted 19 days ago
Similar jobs that could be interesting for youBased on the AI Agent Evaluation Engineer - Remote in San Jose, CA vacancy
$60 per hour
...contributors for a part-time QA project focused on autonomous AI agents. This flexible remote opportunity requires strong analytical and critical... ...with structured data formats. Candidates will review evaluation tasks, identify inconsistencies, and help define expected...Remote workPart timeFlexible hoursMind Rift
Kansas City, MO4 days ago$80 per hour
A technology firm is seeking QAs for autonomous AI agents to validate and improve task structures within a new project. Candidates will... ...thinkers with strong attention to detail and experience in evaluating scenarios. This flexible, project-based role offers competitive...Remote jobFlexible hours$80 per hour
A leading AI consultancy is seeking a detail-oriented individual to design evaluation scenarios for LLM-based agents. This part-time, remote role allows you to create structured test cases while working flexibly around your commitments. The ideal candidate holds a relevant...Remote workPart time$80 per hour
...-time opportunity focused on quality assurance for autonomous AI agents. You will analyze complex systems, review tasks for logic, and... ...analytical and detail-oriented skills, with experience in policy evaluation or logic puzzles preferred. Compensation can reach up to $80/...Remote jobPart timeFlexible hours- ...technology company is looking for a detail-oriented individual to design structured evaluation scenarios for AI agents. This entry-level part-time role allows you to contribute remotely, creating test cases for LLM-based agents. The ideal candidate will have a background...Remote jobPart time
$60 per hour
...ethically shape the future of AI. What We Do The... ...thrive in ambiguity, enjoy remote asynchronous work, and... ...systems are tested and evaluated, we want to hear from... ...experts for autonomous AI agents in a project focused on... ...to LLMs, prompt engineering, or AI‑generated content...Remote workFreelanceFlexible hours$175k - $287k
...the business needs of the team. LinkedIn’s Core AI is building the Evaluation Operating System (EOS), a foundational Agent Evaluation platform that defines how all AI... ...infrastructure for all LinkedIn AI Agents.As a Staff Engineer, you will own the end-to-end technical vision,...For contractorsWork experience placementWork at officeFlexible hours$60 per hour
A leading AI firm in Austin is looking for QA experts to validate... ...and improve AI systems. This remote, freelance role requires... ...detail. Candidates will review AI evaluation tasks, identify inconsistencies... ...define expected behaviors for agents. Ideal applicants have experience...Remote jobFreelance$144.7k - $261.3k
Job DescriptionAbout the TeamThe Evaluation Foundations team, part of Embodied AI's Scaling Foundations, solves critical... ...vehicle development. We engineer high-performance tools that identify... ...more.#GM-AV-1 This role is based remotely, but if the selected candidate...Remote workFull timeLocal areaWork from homeRelocation packageFlexible hours- YO AI Labs seeks an experienced Civil Engineer to contribute to AI evaluation projects by creating realistic civil tasks, applying structural design and regulatory codes, and developing precise solutions. As a contractor, you will produce engineering drawings, reports,...Remote jobFor contractors
- ...partner is looking for a Senior Automation Engineer (Python, AI Agents & Automation) based in United States... ...environments. Benefits ~100% remote work within the applicable location.... ...will process your personal data to evaluate your candidacy and share relevant information...Remote workFull time
jobgether
United States6 days ago $80 per hour
...ethically shape the future of AI. What We Do The Mindrift... ...realistic and structured evaluation scenarios for LLM-based agents. You'll create test cases... ...Science, Software Engineering, Data Science / Data Analytics... ...Take part in a flexible, remote, freelance project that...Remote workPart timeFreelanceFlexible hours- ...Systems in the United States seeks a dual persona AI Curriculum Developer or AI Solutions Engineer to build hands-on AI agent training, demos, and labs across our platform.... ..., and internal teams. This role offers remote/hybrid flexibility and a competitive base salary...Remote work
- ...Tyndale is expanding our AI and automation capabilities... ...seeking an experienced AI Agent & Automation Engineer to build production solutions... ...support. HYBRID/REMOTE: Tyndale supports a strong... ...platforms, AI agent testing/evaluation, containerization and operational...Remote workCasual work1 day per week
Tyndale
Pipersville, PA18 days ago - ...companies, working with frontier AI labs to accelerate model... ...high-quality training data, evaluations, and engineering talent. We are staffing a... ...to develop and evaluate AI agents. You will be assigned to one... ...edge AI systems. Fully remote opportunity. Collaborate...Remote workFull timeContract workTemporary workFlexible hours
Turing
Remote3 days ago $152k - $241.5k
...into the unlimited potential of AI to define the next era of... ...looking for a Senior Software Engineer to join our DGX Cloud / Fleet Intelligence team and build agent-side systems that collect GPU health... ...SummaryLocation: US, CA, Santa Clara; US, NY, Remote; US, RemoteType: Full timeRemote workFull timeLocal area- ...Role Overview Shape evaluation tasks for AI systems used in enterprise electrical systems and product... ...Create electrical engineering scenarios covering complex circuit and... ...design documentation. Work Terms ~ Remote, hourly engagement. Compensation...Remote workHourly pay
SaidGig
United States10 days ago $70 - $80 per hour
...Role Overview Shape evaluation tasks for AI systems used in large-scale infrastructure and construction contexts. This remote, hourly role focuses on work with significant design complexity... ...Key Responsibilities Create civil engineering scenarios covering infrastructure...Remote workHourly paySaidGig
United States10 days ago$80 - $110 per hour
...enterprise client is seeking experienced Structural Engineers to help train and evaluate next-generation AI systems. This opportunity is part of a large-scale... ...engineering. Work independently in a fully remote, flexible environment. Qualifications Requirements...Remote workHourly payContract work10 hours per weekFlexible hoursLifted, an Upwork Company™
Chicago, IL9 days ago- General Motors, through Embodied AI, seeks a Senior Engineer to measure and visualize AV model performance. You will design and implement evaluation workflows, collaborate across Data, Infra... ...with real-world behavior. This remote role offers opportunities to influence...Remote job
$70 - $80 per hour
...Role Overview Shape high-stakes AI evaluation tasks grounded in enterprise product design... .... You will create realistic mechanical engineering scenarios, reference outputs, and assessment... ...is preferred. Work Terms Remote, hourly engagement. Experience with...Remote workHourly paySaidGig
United States10 days ago$166k - $220k
...of systems is powered by Lattice OS, an AI-powered operating system that turns... ...military in months, not years.The Test & Evaluation Engineering Team at Anduril works across the entire... ...Experience conducting test activities in remote, austere, or tactical operational...Remote workFull timeWork experience placementImmediate start$60 - $90 per hour
...Overview Design rigorous, real-world materials science and engineering tasks that evaluate an AI model’s expert reasoning. You will create prompts,... ...correct and why an incorrect answer fails. Work Terms ~ Remote, hourly engagement. Compensation ~$60 to $90 per...Remote workHourly payWork at officeSaidGig
United States13 days ago- ...Join a pioneering AI initiative focused on building... ...the next generation of evaluation benchmarks for frontier... ...QA and Test Engineers to ensure every benchmark... ...integrity. This is a fully remote, full-time engagement requiring... ...language models, AI agent workflows, or...Remote workFull timeContract workFor contractorsFlexible hours
Weekday
Remotea month ago $90 - $175 per hour
...software quality assurance expertise to evaluate technical AI outputs and help improve how next-... ...systems learn, reason, and perform. This remote contract opportunity supports a high-... ...Qualifications Professional experience as a QA Engineer, SDET, Test Engineer, QA Analyst, or...Remote workHourly payContract workSaidGig
United States5 days ago- ...AI Agent Power User - Energy Sector Workflow Evaluation (Remote) is a remote review track for evaluating AI outputs across ai agent power user specialist operations workflows. Reviewers grade workflow correctness, policy adherence, and stakeholder fit; flag operational...Remote jobHourly payFor contractorsWork experience placement10 hours per week
$80 - $100 per hour
...and improve next-generation AI coding agents. In this contractor role you... ...are used for complex software engineering work, shaping model behavior... ...development. Critically evaluate AI agents in diverse... ...type: Contractor. Location: Remote. Work will involve contributing...Remote workHourly payFor contractorsSaidGig
United States2 days ago- ...AI Agent Interaction Specialist - Dialogue & Conversation Evaluation is a remote evaluation track for reviewing ai agent interaction evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind...Remote jobHourly payFor contractors10 hours per week
$180k - $225k
About Scale AIScale AI is the data foundation for AI, helping... ...telecommunications to build production AI agents that automate complex... ...remains one of the hardest engineering challenges.As a Frontier... ...enterprise customers to design, evaluate, and deploy intelligent...Full time- ...IDR is seeking a Software Engineer - AI Evaluation & Automation to join one of our top clients for a remote opportunity. This role is within the technology sector, focusing... ...workflows. Some familiarity with AI, LLM, or agent evaluation processes, including understanding...Remote workTemporary work
IDR
Santa Clara, CA2 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Evaluation Engineer - Remote. Be the first to apply!
Related searches
- executive protection agent San Jose, CA
- cruise agent San Jose, CA
- state farm agent San Jose, CA
- work from home chat agent San Jose, CA
- agent San Jose, CA
- airport agent San Jose, CA
- commissioning agent San Jose, CA
- tsa agent San Jose, CA
- import export agent San Jose, CA
- freight agent no experience San Jose, CA