AI Agent Evaluation Engineer - Remote
YO AI Labs
Job DescriptionJob DescriptionSenior Software Engineer Job Type: Contractor (~15 hours/week) Location: RemoteJob Summary We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required.Key Responsibilities Create reinforcement learning environments for software engineering tasks.Design tasks involving bug fixing, feature development, refactoring, and performance optimization.Build deterministic verification systems and golden reference solutions.Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.Develop realistic, reproducible software engineering scenarios.Ensure tasks accurately measure coding ability, problem-solving, and tool usage.Document solutions and provide clear technical feedback.Required Skills Strong proficiency in Python 3, Java, Rust, C++, or TypeScript.Strong understanding of algorithms and data structures.Experience with bug fixing and debugging complex software issues.Proven experience in feature implementation and codebase refactoring.Strong knowledge of performance optimization and tuning.Excellent written and verbal communication.Strong attention to detail.Preferred Qualifications Experience working with large-scale or distributed codebases.Familiarity with AI/ML systems is a plus but not required.Experience with rigorous code reviews and software engineering best practices.Experience working effectively in remote or cross-functional teams.Hiring Process Submit an application and screening questions.Complete an AI interview (~30 minutes).Complete a technical assessment, if required.Hiring Manager review.Compensation Compensation is output-based, with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.Availability Selected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding.
$80 per hour
...domain experts with cutting‑edge AI projects from innovative tech... ...all security researchers, engineers, and penetration testers with... ...tools for running and evaluating agent behavior. You’ll implement base... ...needs Take part in a flexible, remote, freelance project that fits...Remote workFreelanceFlexible hoursMind Rift
Jackson, MS4 days ago$50 per hour
...seeking contributors to build and evaluate autonomous agents like OpenClaw. This freelance opportunity offers remote work, with pay up to $50 USD... ...help shape the future of AI agents by designing... ...candidates have 2+ years in backend engineering, AI automation, or complex...Remote workWeekly payFreelanceOutlier
Minneapolis, MN4 days ago$50 per hour
...Outlier is inviting skilled engineers to join as OpenClaw Agent Builders on a remote, freelance basis. Earn up to $50 USD... ...incentive options, including access to AI tools and flexible scheduling.... ...with top AI projects, designing, evaluating, and refining agent behaviors in...Remote workHourly payWeekly payFreelanceFlexible hoursOUTLIER
San Diego, CA4 days ago- ...Tyndale is expanding our AI and automation capabilities... ...seeking an experienced AI Agent & Automation Engineer to build production solutions... ...support. HYBRID/REMOTE: Tyndale supports a strong... ...platforms, AI agent testing/evaluation, containerization and operational...Remote workCasual work1 day per week
Tyndale
Abbeville, GA21 days ago - ...landscape, virtual recruiting and remote work are critical for the future of... ...job for you. Position: Agentic QA Engineer Generative AI & Agentic Systems (Agent, Multi-Agent Testing) Location:... ...and fixtures.Experience with LLM evaluation (exact/soft match, BLEU/ROUGE, BERTScore...Remote workPermanent employmentContract workNight shift
Tekfortune Inc
Dallas, TX2 days ago - ...To advance the evaluation and development of frontier coding agents, the full-time remote Member of Technical Staff, Coding Agent Evaluation... ...across various software engineering tasks Analyze model behavior... ...engineering, machine learning, AI research, or related technical...Remote workFull time
Virtual Vocations Inc
United States3 days ago - ...AuraOne is seeking an AI Agent Evaluator to remotely review and label ai agent evaluation prompts and responses against AuraOne's quality rubric. You will compare outputs, identify edge cases, and provide structured feedback to support retraining. The role focuses...Remote work
AuraOne
United States1 day ago - ...Square is seeking a Lead Product Manager to own the agent runtime and evaluation layer of Foundry, the AI platform for building, deploying, and operating... ...agents, apps, and code. You will work with a dedicated engineering team to deliver a governed automation lifecycle...Remote work
Juniper Square
United States3 days ago $60 per hour
...ethically shape the future of AI. What We Do The... ...Does an asynchronous, remote, flexible opportunity sound... ...systems are tested and evaluated? This is a flexible,... ...QAs for autonomous AI agents for a new project focused... ...Exposure to LLMs, prompt engineering, or AI-generated...Remote workPermanent employmentPart timeFreelanceFlexible hoursMind Rift
Raleigh, NC1 day ago- ...Zekai is seeking an AI Agent Evaluator to work remotely from the United States on a contractor basis. You will run real-life scenarios with two AI agents, screen record and score each run, and provide clear written feedback. A Bachelor's degree and the ability to operate...Remote workFor contractors
Zekai
New York, NY5 days ago $25 - $45 per hour
...Role Title: AI Agent Evaluator Role Type: Contractor Location: U.S based Opportunity Snapshot... ...-3 years of experience, preferably in engineering, business management, or office/operations... ...get built and improved This is a 100% remote contract role. Applicants must be...Remote workContract workFor contractorsWork at officeWeekend workConfidential
New York, NY5 days ago$20 - $40 per hour
...seeking detail-oriented contributors to craft realistic AI task scenarios and evaluate agent performance. You will describe steps, judge outcomes, and... ...to help developers improve automation. The role is remote, part-time, hourly, 10–40 hours per week, paid weekly in...Remote workHourly payWeekly payPart time10 hours per weekVettedlance
United States1 day ago- ...HELP SHAPE THE FUTURE OF AI BY EVALUATING HOW WELL AI ASSISTANTS UNDERSTAND USERS, USE PERSONAL CONTEXT, AND DELIVER HELPFUL CONVERSATIONS. Location: Remote Type: Independent contractor / Part-time Project duration: up to 3 months ABOUT THE ROLE At Anyone AI, we create...Remote workPart timeFor contractors
OpenKrill
New York, NY1 day ago $150k - $200k
...US - Remote Full time JR3912 About Us Since 1989, SHI... ...Summary The Solutions Engineer – Agents & Automation is responsible... ...sits at the intersection of AI innovation, human experience... ..., orchestration, grounding, evaluation, tool and function integration...Remote workFull timeWorldwideFlexible hoursSHI GmbH
Albany, NY4 days ago- ...Foley is seeking a GTM Forward Deployed Engineer to advance an AI-native go-to-market engine. You will design and ship experiments that move pipeline, conversions, and cost per lead, building agents that know accounts before reps do. You will own lead scoring, routing...Remote work
Foley
Hartford, CT1 day ago $80 per hour
...A tech company specializing in AI is seeking QAs for autonomous AI agents to validate and enhance task structures... ...allows contributors to work remotely while engaging in a complex AI project... ...detail, facilitating AI testing and evaluation without needing a coding...Remote workFlexible hoursMind Rift
Houston, TX1 day ago- ...technology company is looking for a detail-oriented individual to design structured evaluation scenarios for AI agents. This entry-level part-time role allows you to contribute remotely, creating test cases for LLM-based agents. The ideal candidate will have a background...Remote jobPart time
$60 per hour
...contributors for a part-time QA project focused on autonomous AI agents. This flexible remote opportunity requires strong analytical and critical... ...with structured data formats. Candidates will review evaluation tasks, identify inconsistencies, and help define expected...Remote jobPart timeFlexible hours- ...DataAnnotation is seeking an experienced Electrical Engineer to evaluate how frontier AI models handle real electrical and electronics engineering work. You bring judgment from years designing, debugging, and shipping hardware, while we provide engineering problems the...Remote work
DataAnnotation
United States3 days ago - ...Lifted, an Upwork Company™, is seeking experienced Structural Engineers to help train and evaluate next-generation AI systems. This fully remote, contract opportunity focuses on building AI training tasks around structural engineering concepts using OpenSees and related...Remote workContract work
Lifted, an Upwork Company™
Chicago, IL5 days ago - ...is seeking a Machine Learning Engineer to stress-test how models reason... ...write prompts that probe training, evaluation, debugging, and productionizing ML systems, review AI outputs for subtle errors, and... ...world practitioner experience. Remote, project-based work with strong...Remote work
DataAnnotation
Madison, SD5 days ago - ...DescriptionSenior Software Engineer Job Type: Contractor (~... ...to support an AI training project by creating... ...learning environments that evaluate AI models on complex... ...solutions.Evaluate AI agents' ability to reason through... ...working effectively in remote or cross-functional...Remote workFor contractors
YO AI Labs
Houston, TX2 days ago $50 per hour
...is seeking a freelance OpenClaw Agent Builder to shape the future of... ...autonomous agents. You will work remotely, earning up to $50 USD per hour... ...and collaborating with leading AI projects. Ideal candidates bring 2+ years in backend engineering or AI automation, a track...Remote workHourly payWeekly payFreelanceFlexible hoursOutlier
Chicago, IL4 days ago$30 - $60 per hour
...Tooling Engineer Business Analysis AI Evaluation Evaluate large language models with business-analysis expertise by writing demanding prompts, testing edge cases, and reviewing AI responses. This remote contract role requires a relevant master's degree, three years of...Remote workHourly payContract workPart timeOpenTrain AI, Inc.
New York, NY3 days ago$80 - $110 per hour
...enterprise client is seeking experienced Structural Engineers to help train and evaluate next-generation AI systems. This opportunity is part of a large-scale... ...structural engineering. Work independently in a fully remote, flexible environment. Qualifications Requirements:...Remote workHourly payContract work10 hours per weekFlexible hoursLifted, an Upwork Company™
Chicago, IL4 days ago- ...Job Description Job Description Remote / WFO / Hybrid: Remote Hire Type (... ...02/2026 Rate – 60/hr Backend & Agent Engineer – AI/Agentic Automation Experience: 6–9... ...API, integration, regression, and agent-evaluation tests. Implement logging, tracing,...Remote workContract work
PeopleNTech LLC
Santa Clara, CA15 days ago - ...Founding AI Engineer, Agents & Evaluation New York City Full time On-site Engineering We're building the intelligence layer for all home-based... ...system. // The process – Intro call (45 min, remote): Daniel walks you through the role and the reactive-trap...Remote workPermanent employmentFull timeWork at officeNight shift
Zingage
New York, NY5 days ago - ...DataAnnotation is hiring a Machine Learning Engineer to stress-test model reasoning about ML systems, evaluation design, data pipelines, and deployment trade-offs. You’ll write prompts and review AI outputs, then craft correct solutions grounded in real practitioner experience...Remote work
DataAnnotation
Arden Hills, MN4 days ago $50 per hour
...Outlier helps the world’s most innovative companies improve their AI agents by providing human feedback. Do you want to shape the future of... ..., and optimize complex, real-world workflows. This freelance, remote role offers pay up to $50 USD/hr with weekly payments. #J-18808...Remote workWeekly payFreelanceOutlier
Columbus, OH4 days ago- ...DEFCON AI, Inc. is seeking a Model Test & Measurement Engineer to lead evaluation frameworks for an AI-assisted platform in a secure government cloud. You will build ground... ...influence release decisions. This is a fully remote role with occasional travel to headquarters or...Remote work
DEFCON AI, Inc.
McLean, VA2 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Evaluation Engineer - Remote. Be the first to apply!
