Software Engineer - AI Code Evaluation
$75 - $105 per hourBraintrust
About the Opportunity
Review Python code, tests, and execution logs from AI coding tasks to determine why a solution failed and document the evidence behind your conclusion.
You’ll work with a leading AI research organization to evaluate how coding agents perform on software engineering tasks. A failing test might point to a bug in the agent’s code, an overly restrictive assertion, an unclear requirement, or a problem with the test environment. Your job is to trace what happened and explain whether it affected the result.
What You’ll Do
- Read unfamiliar code and trace an AI coding agent’s changes through the task requirements, test suite, and run logs.
- Reproduce failures and identify what caused them. Check whether the implementation, test assertion, task specification, or environment is responsible.
- Diagnose test harness and environment failures using Docker configuration, dependency information, and execution logs.
- Determine whether the issue changed the task’s pass/fail result. Write a concise finding that cites the relevant code, test, requirement, or log and explains how you verified it.
- Join daily discussions with research scientists to compare findings and apply consistent evaluation criteria.
A typical investigation takes 30–60 minutes .
What We’re Looking For
- 2+ years of professional experience writing, reviewing, and debugging software , or a completed STEM master’s degree or PhD with intensive programming and computational research experience.
- Hands-on experience reading and debugging Python in unfamiliar or complex codebases. You can walk through a real failure you investigated, explain what evidence isolated the cause, and describe how you verified the fix.
- Ability to read at least one additional language, such as JavaScript or TypeScript, Go, C++, or Rust .
- Practical experience with Git, Docker, and software build or test environments .
- Experience with a test framework such as pytest, unittest, Jest, or Mocha . You can read test assertions and assess whether they match the documented behavior.
- Clear written and spoken English for explaining technical findings and the evidence behind them.
Nice to Have
- Experience evaluating LLMs or coding agents, including reviewing their code changes, test results, or execution traces.
- Familiarity with coding benchmarks such as SWE-bench or CodeContests.
- Experience with AI reliability evaluation or collaborating with ML researchers.
Engagement Details
- Location — Remote, United States only
- Expected commitment — 35 hours per week
- Start — Immediate
- Compensation — $75–$105 USD per hour, depending on relevant skills and experience
Interview Process
Shortlisted candidates will need to complete a 15-minute AI interview.
$162k - $260k
...looking forWe’re searching for a Senior Software Engineer, ML Evaluation & Introspection Infrastructure to... ...agent-first: our tools are built for AI agents to drive as easily as engineers... ...alerting, SLOs), and everyday use of AI coding agents.Coming from backend or data...SuggestedWork at officeLocal area3 days per week$175k - $263.3k
Role Number: 200664455Team: Software and ServicesLocation:... ...trustworthy as the methods used to evaluate them. At Apple, where AI powers experiences for... .... Join Apple Services Engineering to build the next generation... ...operationalize scientific code.Exceptional communication...SuggestedRelocation$159.8k - $235k
About the TeamThe Code Quality team sits within the Developer Platform... ...is to make sure the signals engineers rely on before shipping — test... ...the RoleWe're looking for Software Engineers to help build and maintain... ..., and rolloutsExplore how AI agents and tools can automate...SuggestedHourly payWork at officeLocal areaRemote workFlexible hours- Lead by example, driving engineering excellence across the team while... ...s goal, roadmap, and broader AI strategy.Stay ahead of the evolving... ...including Google's software supply chain integrity program... ...and their users safe.The Safe Coding team contributes to that mission...Suggested
- ...research accelerator for frontier AI labs and a trusted partner... ...who specialize in software engineering, logical reasoning, STEM, multilinguality... ...As a Software Engineering evaluator, you will create cutting-... .... This includes curating code examples, providing precise...SuggestedRemote jobFor contractorsFlexible hours
$205.4k - $308.5k
...-04-21SummaryJoin Apple Services Engineering to build the next generation of AI evaluation systems. We are seeking a staff machine... ...Qualifications8+ years of software engineering experience with a track... ..., but you can read research code and assess: is this a software problem...Relocation- ...most productive place of an engineer’s career. The Code Review team builds the... ...judgment.Our goal is to build an AI review agent that... ...Stripe.What you’ll doAs a Staff Software Engineer, you will take ownership... ...pull requestsDevelop evaluations that compare agent and human...
$104k - $166k
...housing relocation costs. As a Software Development Engineer Intern at Zillow Group, you... ...real problems and ship your code. Work at one of our highly... ...Hands-on experience using AI tools to build software, with judgment to evaluate and refine AI-generated output...Permanent employmentSummer workInternshipSummer internshipLive inWork at officeLocal areaRemote workRelocation- ...companies, working with frontier AI labs to accelerate model... ...high-quality training data, evaluations, and engineering talent. About the Role... ...looking for experienced, hands-on software engineers to help evaluate and improve AI coding models . Rather than...Remote jobHourly payFor contractorsImmediate start
$100k - $170k
...Artly AI | Seattle, WA | Full-time | On-site... ...builds robots and the software that makes them useful... ...a Full Stack Software Engineer who can independently build... ...operations. • Use AI coding assistants and agentic... ...reliable delivery. We do not evaluate AI proficiency by lines...Full timeLocal area$150.4k - $277.6k
...Number: 200690248Team: Software and ServicesLocation: Cupertino... ...for a senior software engineer to build the next... ..., the backbone of Siri AI and Apple Intelligence.... ...and security teams to evaluate techniques such as cache... ...write high performance code. You'll also build...Remote workWorldwideRelocation$148.5k - $223.9k
...SalesforceSalesforce is the #1 AI CRM, where humans with... ....Our Public Cloud engineering teams are responsible... ...Developing Infrastructure-as-Code using TerraformWriting... ...4+ years backend software development experienceDeep... ...recruiters assess and evaluate candidates’ resumes and...Full time$159.8k - $235k
...reliable on-demand, logistics engine for delivery! We're looking for... ...-functionally, as well as evaluating and executing on trade-offs to... ...tech stacksProficiency in using AI coding tools (e.g., Claude Code, Codex, Cursor) in the full software development lifecycle, including...Hourly payWork at officeLocal areaRemote workFlexible hours$180k - $230k
...looking for a senior full stack engineer with a backend lean who... ...rely on, and who builds with AI coding agents as a natural part of... ...with their clients, and the evaluation systems that keep our AI accurate... ...: 6+ years of software engineering experience, including...- ...reliability, performance, and engineering quality.- Collaborate... ....- 5+ years of software development experience,... ...systems.- Experience using AI-assisted development tools... ...productivity, with the ability to evaluate, validate, and debug AI-generated code or solutions.- Strong...
$148.5k - $223.9k
...SalesforceSalesforce is the #1 AI CRM, where humans with... ...ExperienceThe MeshMesh Engineering team is building the... .... As a Front End Software Engineer, you will build... ...process, culture, and code base, then improve it.Design... ...recruiters assess and evaluate candidates’ resumes and...Full time$130.6k - $192k
...reliable on-demand, logistics engine for delivery! We're looking for... ...-functionally, as well as evaluating and executing on trade-offs to... ...tech stacksProficiency in using AI coding tools (e.g., Claude Code, Codex, Cursor) in the full software development lifecycle, including...Hourly payWork at officeLocal areaRemote workFlexible hours$170k - $210k
...we're changing that. We're an AI-native benefits intelligence... ...We’re looking for a Senior Software Engineer who thrives in a hands-on role... ...writing a lot of production code across our TypeScript-based stack... ...Beat Yesterday. We re-evaluate our own toolkit every quarter...Work at officeRemote workNight shift- ...As a Senior Software Engineer , you will design and build scalable backend... ...and reliable integrations with AI technologies. This role... ...Write well-tested, maintainable code using strong unit and... ...source LLM models . Develop evaluation approaches such as A/B testing...Flexible hours
$142.3k - $263.3k
Role Number: 200686871Team: Software and ServicesLocation:... ...CDI) is the central data, AI, and machine learning engineering team in ASE Media. We build... ...skills and agents are built, evaluated, shared, and maintained... ...systemsExperience with infrastructure-as-code systems such as Terraform...Work at officeRelocation$184.5k
...traveler satisfaction.The Senior Software Development Engineer - ML Platform role is a... ...integrate and operate AI/ML-enabled solutions that improve... ...design, low-level design, coding, testing, debugging, and... ...delivery, with experience evaluating tradeoffs, managing reliability...Full time$180k - $225k
...Role:We are hiring a Senior Software Engineer to join Global Network Experience... ...agent interfaces that let AI do the same under real change... ...partner engineering teams.Design, code, and test services, APIs,... ...into project estimates, and evaluate AI-generated code for correctness...Full timeWork at officeImmediate startWorldwideFlexible hoursShift work- ...world.The role:As a Senior Software Engineer (IC3) on SoFi’s Prism team,... ...build Prism, SoFi’s internal AI platform for knowledge work.... ...partner teams (such as Autonomous Coding, CI/CD, and product squads),... ...architectures (RAG), and AI evaluation frameworks.Compensation and...Remote work
$148.5k - $223.9k
...duplicating efforts. Job Category Software Engineering Job Details About Salesforce Salesforce is the #1 AI CRM, where humans with agents... ...Developing Infrastructure-as-Code Writing microservices on... ...help our recruiters assess and evaluate candidates' resumes and qualifications...Work experience placement$177k - $216k
...are Metropolis is seeking a Senior Software Engineer to architect and build our AI agent swarm, orchestration... ...and token usage Establish robust evaluation, tracing, and observability frameworks... ...development tools such as Claude Code, GitHub Copilot, or similar agentic...Temporary workWork at officeLocal area- ...scientific discovery to powering AI and the technologies people... ...are seeking an experienced software systems engineer to join AMD's Datacenter... ...organization through design reviews, code reviews, troubleshooting... ...drive timely resolution. Evaluate emerging technologies and...
$160k - $210k
Senior Software Engineer, Backend Bellevue, WA Are you ready to revolutionize... ...will be at the forefront of AI-driven advertising solutions,... ...the team through high-quality code reviews, technical mentorship... ...do not influence candidate evaluation or hiring decisions. Cognitiv...Work at officeImmediate startRemote workWork from home- ...an opportunity for a Senior Software Engineer – Ad Intelligence Data Platform... ...advertising through data and AI across streaming and TV... ...including CI/CD, automated testing, code reviews, and system design... ...Researchers, and QA teams. Evaluate and implement emerging...Hourly payContract workLocal area
$116.2k - $229.1k
...027The Work You’ll DoThe Software & ALM Senior Consultant executes... ...-on delivery to build AI-enabled software and systems engineering lifecycle workflows (... ...approvals and auditability.Build evaluation approaches/harnesses to... ...requirements, design, code, test, and release (...Local area$121.6k - $243.2k
...managers, designers, and engineers to translate product... .... - Build or integrate AI-powered product experiences... ...workflows to improve code quality, testing,... ...Understanding of fundamental software engineering concepts,... ...product problems; ability to evaluate, debug, and iteratively...Temporary workLocal area
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Software Engineer - AI Code Evaluation. Be the first to apply!
- senior software design engineer Seattle, WA
- software web developer Seattle, WA
- senior software engineer Seattle, WA
- software developer fintech Seattle, WA
- software support engineer Seattle, WA
- security software engineer Seattle, WA
- software developer night shift Seattle, WA
- scientific software engineer Seattle, WA
- software engineer - web development Seattle, WA
- new graduate software engineer Seattle, WA





