AI Agent Evaluation Engineer - Remote
YO AI Labs
Philadelphia, PA
- Remote job
Job Description
Job Description
Senior Software Engineer
Job Type: Contractor (~15 hours/week)
Location: Remote
We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.
You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization . No prior AI experience is required.
Key Responsibilities- Create reinforcement learning environments for software engineering tasks.
- Design tasks involving bug fixing, feature development, refactoring, and performance optimization .
- Build deterministic verification systems and golden reference solutions.
- Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
- Develop realistic, reproducible software engineering scenarios.
- Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
- Document solutions and provide clear technical feedback.
- Strong proficiency in Python 3, Java, Rust, C++, or TypeScript .
- Strong understanding of algorithms and data structures .
- Experience with bug fixing and debugging complex software issues .
- Proven experience in feature implementation and codebase refactoring .
- Strong knowledge of performance optimization and tuning .
- Excellent written and verbal communication.
- Strong attention to detail.
- Experience working with large-scale or distributed codebases .
- Familiarity with AI/ML systems is a plus but not required.
- Experience with rigorous code reviews and software engineering best practices .
- Experience working effectively in remote or cross-functional teams.
- Submit an application and screening questions.
- Complete an AI interview (~30 minutes).
- Complete a technical assessment, if required.
- Hiring Manager review.
Compensation is output-based , with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.
AvailabilitySelected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding .
Vacancy posted 16 days ago
Similar jobs that could be interesting for youBased on the AI Agent Evaluation Engineer - Remote in Philadelphia, PA vacancy
- ...technology company is looking for a detail-oriented individual to design structured evaluation scenarios for AI agents. This entry-level part-time role allows you to contribute remotely, creating test cases for LLM-based agents. The ideal candidate will have a background...Remote workPart time
Mind Rift
Brooklyn, NY1 day ago $60 per hour
...contributors for a part-time QA project focused on autonomous AI agents. This flexible remote opportunity requires strong analytical and critical... ...with structured data formats. Candidates will review evaluation tasks, identify inconsistencies, and help define expected...Remote workPart timeFlexible hoursMind Rift
Kansas City, MO1 day ago$60 per hour
...A leading AI firm in Austin is looking for QA experts to validate... ...and improve AI systems. This remote, freelance role requires... ...detail. Candidates will review AI evaluation tasks, identify inconsistencies... ...define expected behaviors for agents. Ideal applicants have experience...Remote workFreelanceMind Rift
Austin, TX5 days ago$80 per hour
...-time opportunity focused on quality assurance for autonomous AI agents. You will analyze complex systems, review tasks for logic, and... ...analytical and detail-oriented skills, with experience in policy evaluation or logic puzzles preferred. Compensation can reach up to $80/...Remote jobPart timeFlexible hours$60 per hour
...innovation company is seeking QAs for autonomous AI agents to validate and improve task structures and evaluate logic. The role requires excellent analytical thinking... .... Successful candidates can work flexibly and remotely, earning rates up to $60/hour. This position is...Remote jobFlexible hours$80 per hour
A technology firm is seeking QAs for autonomous AI agents to validate and improve task structures within a new project. Candidates will... ...thinkers with strong attention to detail and experience in evaluating scenarios. This flexible, project-based role offers competitive...Remote jobFlexible hours$80 per hour
A leading AI consultancy is seeking a detail-oriented individual to design evaluation scenarios for LLM-based agents. This part-time, remote role allows you to create structured test cases while working flexibly around your commitments. The ideal candidate holds a relevant...Remote workPart time$60 per hour
...ethically shape the future of AI. What We Do The... ...Does an asynchronous, remote, flexible opportunity sound... ...systems are tested and evaluated? This is a flexible,... ...QAs for autonomous AI agents for a new project focused... ...to LLMs, prompt engineering, or AI-generated content...Remote workPermanent employmentPart timeFreelanceFlexible hours$175k - $287k
...the business needs of the team. LinkedIn’s Core AI is building the Evaluation Operating System (EOS), a foundational Agent Evaluation platform that defines how all AI... ...infrastructure for all LinkedIn AI Agents.As a Staff Engineer, you will own the end-to-end technical vision,...For contractorsWork experience placementWork at officeFlexible hours$200k - $275k
...Secure Every Identity, from AI to Human Identity is the key... ...let's talk. About Okta for AI Agents Okta secures access for 2... ...identity, platform, and security engineering teams, write production code... ...outcomes end to end. #LI-Remote P25257_3456702 Below is...Remote workLocal areaWorldwideFlexible hoursShift workOkta, Inc.
Bellevue, WA4 days ago- ...leading security-first enterprise AI company, is seeking a Member... ...Staff in the Safety for Agents team to advance safer LLMs through... ...post-training algorithms, and evaluation methods. You'll collaborate with... ...across global offices, while remote-friendly policies support...Remote workFlexible hours
$80 per hour
A tech company specializing in AI is seeking QAs for autonomous AI agents to validate and enhance task structures... ...allows contributors to work remotely while engaging in a complex AI project... ...detail, facilitating AI testing and evaluation without needing a coding...Remote jobFlexible hours$55 per hour
A leading AI innovation firm is seeking QAs for autonomous AI agents to improve evaluation frameworks. Candidates should possess excellent analytical thinking and attention... ...tasks and define clear standards. This remote, flexible opportunity offers rates up to $55/...Remote jobPart timeFlexible hours- ...partner is looking for a Senior Automation Engineer (Python, AI Agents & Automation) based in United States... ...environments. Benefits ~100% remote work within the applicable location.... ...will process your personal data to evaluate your candidacy and share relevant information...Remote workFull time
jobgether
United States3 days ago $80 per hour
A leading AI consultancy in the United States is seeking Quality Assurance professionals to validate and improve autonomous AI agents. You will analyze complex systems, evaluate task structures, and ensure logical consistency through detailed reviews. Ideal candidates possess...Remote jobHourly payFlexible hours$144.7k - $261.3k
Job DescriptionAbout the TeamThe Evaluation Foundations team, part of Embodied AI's Scaling Foundations, solves critical... ...vehicle development. We engineer high-performance tools that identify... ...more.#GM-AV-1 This role is based remotely, but if the selected candidate...Remote workFull timeLocal areaWork from homeRelocation packageFlexible hours- YO AI Labs is seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using MCP tools. You will design reproducible environments, deterministic...Remote job
$80 per hour
...ethically shape the future of AI. What We Do The Mindrift... ...realistic and structured evaluation scenarios for LLM-based agents. You'll create test cases... ...Science, Software Engineering, Data Science / Data Analytics... ...Take part in a flexible, remote, freelance project that...Remote workPart timeFreelanceFlexible hours$55 per hour
A leading AI innovation firm in Dallas is seeking QAs for autonomous AI agents to ensure the quality of complex systems and scenarios. This flexible, remote project is ideal for those with excellent analytical... ...should be adept at evaluating scenarios and documenting findings...Remote jobFlexible hours- YO AI Labs seeks experienced Civil Engineers for a remote contractor role contributing to AI evaluation. You will craft realistic engineering tasks, generate supporting drawings and reports, and assess code compliance across structural, transportation, and water systems...Remote jobFor contractors
- YO AI Labs is seeking experienced Civil Engineers to contribute to an AI evaluation project. You will apply your engineering expertise to create realistic technical scenarios, develop accurate solutions, and evaluate AI-generated outputs. No prior AI experience is required...Remote job
- YO AI Labs is seeking experienced civil engineers to contribute to an AI evaluation project. You will apply engineering expertise to create realistic scenarios, develop solutions... ...on code compliance and technical accuracy. Remote-friendly engagement with quick onboarding and...Remote job
- YO AI Labs is seeking experienced Civil Engineers for a remote, output-based evaluation project focused on AI-driven engineering tasks. You will apply engineering expertise to create realistic scenarios, develop accurate solutions, and assess AI-generated outputs. No prior...Remote job
- YO AI Labs seeks an experienced Civil Engineer to contribute to AI evaluation projects by creating realistic civil tasks, applying structural design and regulatory codes, and developing precise solutions. As a contractor, you will produce engineering drawings, reports,...Remote jobFor contractors
- ...in the United States seeks experienced Mechanical Engineering professionals to contribute to AI evaluation and training projects. You will shape enterprise-grade... ..., and regulatory processes in Fortune 500 environments. Remote, flexible schedule. #J-18808-Ljbffr SynthiresRemote jobFlexible hours
- ...Tyndale is expanding our AI and automation capabilities... ...seeking an experienced AI Agent & Automation Engineer to build production solutions... ...support. HYBRID/REMOTE: Tyndale supports a strong... ...platforms, AI agent testing/evaluation, containerization and operational...Remote workCasual work1 day per week
Tyndale
Pipersville, PA15 days ago - Andrews Recruitment Group is seeking experienced Mechanical Engineering AI Experts for remote contractor work on a large-scale project. The role focuses on evaluating AI-generated engineering outputs, solving mechanical problems, and creating rigorous assessments. Background...Remote jobFor contractorsFlexible hours
- YO AI Labs in the United States is seeking experienced Civil Engineers to contribute to a high-impact AI evaluation project. You will apply your engineering expertise to create realistic scenarios, develop accurate solutions, and evaluate AI-generated engineering outputs...Remote jobFor contractors
$152k - $241.5k
...into the unlimited potential of AI to define the next era of... ...looking for a Senior Software Engineer to join our DGX Cloud / Fleet Intelligence team and build agent-side systems that collect GPU health... ...SummaryLocation: US, CA, Santa Clara; US, NY, Remote; US, RemoteType: Full timeRemote workFull timeLocal area- YO AI Labs is seeking experienced Civil Engineers for remote, contractor-type work focused on AI evaluation. You will craft realistic civil engineering tasks, prepare technical materials, and assess code compliance across design domains. The role emphasizes constraint...Remote jobFor contractors
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Evaluation Engineer - Remote. Be the first to apply!
Related searches
- executive protection agent Philadelphia, PA
- cruise agent Philadelphia, PA
- telemarketer - state farm agent team member Philadelphia, PA
- state farm agent Philadelphia, PA
- work from home chat agent Philadelphia, PA
- agent Philadelphia, PA
- airport agent Philadelphia, PA
- commissioning agent Philadelphia, PA
- operations agent Philadelphia, PA
- tsa agent Philadelphia, PA