Software Engineer - AI Code Evaluation
$75 - $105 per hourBraintrust
About the Opportunity
Review Python code, tests, and execution logs from AI coding tasks to determine why a solution failed and document the evidence behind your conclusion.
You’ll work with a leading AI research organization to evaluate how coding agents perform on software engineering tasks. A failing test might point to a bug in the agent’s code, an overly restrictive assertion, an unclear requirement, or a problem with the test environment. Your job is to trace what happened and explain whether it affected the result.
What You’ll Do
- Read unfamiliar code and trace an AI coding agent’s changes through the task requirements, test suite, and run logs.
- Reproduce failures and identify what caused them. Check whether the implementation, test assertion, task specification, or environment is responsible.
- Diagnose test harness and environment failures using Docker configuration, dependency information, and execution logs.
- Determine whether the issue changed the task’s pass/fail result. Write a concise finding that cites the relevant code, test, requirement, or log and explains how you verified it.
- Join daily discussions with research scientists to compare findings and apply consistent evaluation criteria.
A typical investigation takes 30–60 minutes .
What We’re Looking For
- 2+ years of professional experience writing, reviewing, and debugging software , or a completed STEM master’s degree or PhD with intensive programming and computational research experience.
- Hands-on experience reading and debugging Python in unfamiliar or complex codebases. You can walk through a real failure you investigated, explain what evidence isolated the cause, and describe how you verified the fix.
- Ability to read at least one additional language, such as JavaScript or TypeScript, Go, C++, or Rust .
- Practical experience with Git, Docker, and software build or test environments .
- Experience with a test framework such as pytest, unittest, Jest, or Mocha . You can read test assertions and assess whether they match the documented behavior.
- Clear written and spoken English for explaining technical findings and the evidence behind them.
Nice to Have
- Experience evaluating LLMs or coding agents, including reviewing their code changes, test results, or execution traces.
- Familiarity with coding benchmarks such as SWE-bench or CodeContests.
- Experience with AI reliability evaluation or collaborating with ML researchers.
Engagement Details
- Location — Remote, United States only
- Expected commitment — 35 hours per week
- Start — Immediate
- Compensation — $75–$105 USD per hour, depending on relevant skills and experience
Interview Process
Shortlisted candidates will need to complete a 15-minute AI interview.
$90 per hour
...Pay: Up to $90+/hr Experienced software engineers review and score AI-written code for frontier AI research teams. Remote and flexible, up to 40 hours a week. Pay is up to $90+ per hour for the most experienced engineers. Requirements: several years of professional...SuggestedHourly payTemporary workFor contractorsLive inRemote workVisa sponsorshipFlexible hours$174k - $252k
...infra blocks, such as streaming evaluation, autorater pipelines, and... ....5 years of experience with coding in Python.3 years of... ...experience working with Generative AI agents.2 years of experience... ...in Python and C++.Google's software engineers develop the next-generation...Suggested$175k - $315k
...world’s most advanced AI-native platform. We work... ...CrowdStrike Cloud Detection Engine (CDE) team is building... ...engine that evaluates trillions of events a day... ...paradigms to effective peer code reviews and resilient architecture... ...produce high-quality software that is unit tested,...SuggestedPermanent employmentWork experience placementWork at officeLocal area2 days per week3 days per week$136k - $245k
...with the world’s most advanced AI-native platform. We work on... ...development process, so vulnerable code is identified as it is written... ...secure applications. Our engineers work with AI coding agents... ...through to resolution. Keep re-evaluating our architecture, test...SuggestedPermanent employmentWork experience placementWork at officeLocal area3 days per week$197.3k - $225.1k
AI Engineer 4 (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) At Capital One, we are creating responsible and reliable AI... ...test, deploy, and support AI software components including foundation... ...Francisco, California Police Code Article 49, Sections 4901-4920...SuggestedFull timePart timeLocal area$229.9k - $262.4k
AI Engineer 5 (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) At Capital One, we are creating responsible and reliable AI... ...test, deploy, and support AI software components including foundation... ...Francisco, California Police Code Article 49, Sections 4901-4920...Full timePart timeLocal area$229.9k - $262.4k
Sr. Manager, AI Engineer (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) At Capital One, we are creating responsible and... ..., deployment, and support AI software components including foundation... ...Francisco, California Police Code Article 49, Sections 4901-4920...Full timePart timeLocal area$164k - $295k
...world’s most advanced AI-native platform. We work... ...directly with product engineering teams and their leadership... ...— writing production code, re‑architecting critical... .... Continuously re‑evaluate our products to improve... ...community; evangelize software engineering best practices...Permanent employmentWork experience placementWork at officeLocal area2 days per week3 days per week$159.8k - $235k
About the TeamThe Code Quality team sits within the Developer Platform... ...is to make sure the signals engineers rely on before shipping — test... ...the RoleWe're looking for Software Engineers to help build and maintain... ..., and rolloutsExplore how AI agents and tools can automate...Hourly payWork at officeLocal areaRemote workFlexible hours- ...Financial Services & InsuranceAI Evaluation Engineer PRIMARY PURPOSE: The AI Evaluation Engineer designs, builds... ...preferred. ~ Experience ~4+ years in software, ML, or data engineering, with at... ...(testing, version control, CI/CD, code review). ~ Practical experience...Remote work
$121.79k - $166.95k
...have an exciting opportunity to join our team as a Lead Software Engineer - AI Coding. As a Lead Software Engineer in AI Coding, you will be responsible... ...software products based on published maturity model Evaluate tools and technologies and provide analysis and...Full timeContract workFlexible hours$220k - $300k
...organizations use to build their own clinical AI agents. Our mission is to make... .... About this role As a Software Engineer, Voice Evaluation at Concurrence, you'll build the system... ...through to a verified fix You use coding agents in your work and independently...Full timeFlexible hours- ...Senior Software Engineer - Job Description Location: Remote (Anywhere in the USA) We are... ...Software Engineer who thrives on writing code and pushing projects to production... ...Best Practice Evolution: Research, evaluate, and implement modern software engineering...Full timeContract workImmediate startRemote work
- ...Company Cassidy is a no-code platform that allows... ...businesses to build custom AI assistants that... ...faster without heavy engineering overhead. About the... ...years of professional software development experience... ...AI : Familiarity with evaluating AI models or integrating...
$164k - $295k
...modern security with the world’s most advanced AI-native platform. We work on large scale... ...'ll harness cutting-edge technologies to engineer robust backend services that seamlessly... ...experience including testing strategies, code reviews, and resilient architecture design...Permanent employmentWork experience placementWork at officeLocal areaWorldwide2 days per week3 days per week$10,500 per month
...builds the world's leading software for data-driven decisions... ...The Role As a Software Engineer Intern on Sovereign AI Infrastructure, you will help... ...identity or permissions, evaluations, and the core workflows... ...looking for interns with strong coding fundamentals and...Full timeWork experience placementInternshipWork at officeRemote workWork from home$125k - $225k
...modern security with the world’s most advanced AI-native platform. We work on large scale... ...Relentlessly pursue quality by following engineering best practices including: Appropriate testing paradigms, effective peer code reviews, logging, monitoring, and resilient...Permanent employmentWork experience placementWork at officeLocal area3 days per week- ...Senior Software Engineer, Agent Platform New York, NY A recently-minted... ...investors — is building the AI infrastructure powering human... ...autonomous agents get built, evaluated, and trusted in production, inventing... ...assisted development (Claude Code, Cursor, or similar) as a...Full time
$157k - $235k
...digital services.Snap Engineering teams build fun and technically... ...product requirements, evaluate trade-offs, and deliver... ...cost managementUtilize AI tools and high velocity... ...rigorous standards for code correctness, security,... ...of post-Bachelor’s software development experience;...Full timeLive inWork at officeLocal area$180k - $260k
...modern platforms and AI, you’ll do the most meaningful... ...is the innovation and engineering hub behind Epiq’s next... ..., retrieval, and evaluation depend on.Instrument... ...-call rotation.Use AI coding agents throughout development... ...delivering working software.Required...Full timeWork experience placementLocal area- ...Senior Software Engineer | AI-First Development New York City – Financial District 4 Days/Week... ...Windsurf to accelerate development, improve code quality, automate workflows, and solve... ...productivity. Continuously evaluate emerging AI development tools and techniques...Relocation
$124.9k - $167.4k
...global organization of engineers, product developers, designers... ...Job Summary As a Software Engineer II, you will... ...and build intelligent, AI driven systems that... ...should be measured and evaluated for reliability and compliance... ...(e.g., Cursor, Claude Code) to accelerate...- ...Full Stack Engineer NYC MUST BE ELIGIBLE FOR SECURITY CLEARANCE... ...Key Responsibilities Evaluate, refine, and improve AI-powered search and review... ..., and maintainable code while contributing to a collaborative... ...~3+ years of professional software engineering experience....Full time
$164k - $295k
...modern security with the world’s most advanced AI-native platform. We work on large scale... ...We're looking for a Senior UI Platform Engineer to lead the design and build of the shared... ...scaffold, build, integrate and ship their code across the organisation. UI Platform owns...Permanent employmentContract workWork experience placementWork at officeLocal areaRemote workShift work- ...distress. They are seeking a full-time Software Engineer (Data), AI Platform to work fully onsite out of... ...generation, vector search, and evaluation frameworks • Proficiency with AI-assisted... ...development tooling (e.g., Claude Code, Cursor) as a primary development...Full timeWork at office
- ...Senior Software Engineer – AI | Forward Deployed Engineering Location: Hybrid, with required office... ...usage, and system health. Develop evaluation pipelines and benchmarks to assess... ..., Azure, or GCP and infrastructure as code tools such as Terraform or CDK. ~ Experience...Full timeContract workWork at office
$176k - $191k
...TeamDeveloper Experience & AI Velocity is a new team... ...experience for Wonder Engineering — 400+ engineers... ...and controlled costs.AI coding tools are now a... ...testing, and debugging software at Wonder, but adoption... ...with an objective lens — evaluating people for their potential...Full timeTemporary workWork at officeFlexible hours3 days per week$117.7k - $218.6k
Senior Software Engineer, Search Platform Our vision We are building the next... ...platform. Job DescriptionThe AI Enabled Software Engineering... ...retrieval components so offline evaluation and LLM-as-judge grading can... ...) and infrastructure as code Operational ownership: on-call...Full timeWork at officeLocal areaFlexible hours$141k
...ecosystem of devices and cloud software. Like our products, we... ...infrastructure with an AI intelligence layer—... ...experienced platform engineer to join Prepared's... ...knowledge of infrastructure-as-code tools like Terraform,... ...from applicants to evaluate candidates for employment...Work experience placementWork at office$148.5k - $223.9k
...SalesforceSalesforce is the #1 AI CRM, where humans with agents... ...’re looking for a Senior Software Engineer to take important, poorly defined... ...into what we deliver. Evaluate design choices against enterprise... ...effectiveness of the team through code reviews, clear decisions, and...Full time
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Software Engineer - AI Code Evaluation. Be the first to apply!
- junior software developer New York, NY
- senior software design engineer New York, NY
- software web developer New York, NY
- senior software engineer New York, NY
- software sales engineer New York, NY
- software developer fintech New York, NY
- software support engineer New York, NY
- security software engineer New York, NY
- oracle software engineer New York, NY
- software developer night shift New York, NY








