Software Engineer - AI Code Evaluation
$75 - $105 per hourBraintrust
About the Opportunity
Review Python code, tests, and execution logs from AI coding tasks to determine why a solution failed and document the evidence behind your conclusion.
You’ll work with a leading AI research organization to evaluate how coding agents perform on software engineering tasks. A failing test might point to a bug in the agent’s code, an overly restrictive assertion, an unclear requirement, or a problem with the test environment. Your job is to trace what happened and explain whether it affected the result.
What You’ll Do
- Read unfamiliar code and trace an AI coding agent’s changes through the task requirements, test suite, and run logs.
- Reproduce failures and identify what caused them. Check whether the implementation, test assertion, task specification, or environment is responsible.
- Diagnose test harness and environment failures using Docker configuration, dependency information, and execution logs.
- Determine whether the issue changed the task’s pass/fail result. Write a concise finding that cites the relevant code, test, requirement, or log and explains how you verified it.
- Join daily discussions with research scientists to compare findings and apply consistent evaluation criteria.
A typical investigation takes 30–60 minutes .
What We’re Looking For
- 2+ years of professional experience writing, reviewing, and debugging software , or a completed STEM master’s degree or PhD with intensive programming and computational research experience.
- Hands-on experience reading and debugging Python in unfamiliar or complex codebases. You can walk through a real failure you investigated, explain what evidence isolated the cause, and describe how you verified the fix.
- Ability to read at least one additional language, such as JavaScript or TypeScript, Go, C++, or Rust .
- Practical experience with Git, Docker, and software build or test environments .
- Experience with a test framework such as pytest, unittest, Jest, or Mocha . You can read test assertions and assess whether they match the documented behavior.
- Clear written and spoken English for explaining technical findings and the evidence behind them.
Nice to Have
- Experience evaluating LLMs or coding agents, including reviewing their code changes, test results, or execution traces.
- Familiarity with coding benchmarks such as SWE-bench or CodeContests.
- Experience with AI reliability evaluation or collaborating with ML researchers.
Engagement Details
- Location — Remote, United States only
- Expected commitment — 35 hours per week
- Start — Immediate
- Compensation — $75–$105 USD per hour, depending on relevant skills and experience
Interview Process
Shortlisted candidates will need to complete a 15-minute AI interview.
$150k - $270k
...with the world’s most advanced AI-native platform. We work on... ...CrowdStrike is seeking a Senior Software Engineer to help design and build the... ...is an opportunity to write code that operates at a scale few... ...globally Continuously re-evaluate architecture, performance, and...SuggestedHourly payPermanent employmentFull timeWork experience placementWork at officeLocal areaWorldwide2 days per week3 days per week$143.5k - $212.85k
...spanning all phases of the Software Development Lifecycle (... ...issues, guiding junior engineers, operating with little... ...key contributor to our AI-powered tooling... ...assisted infrastructure code generation, and autonomous... ...engineering workflows, including evaluation, safety, and...SuggestedFull timeWork at officeLocal areaImmediate startFlexible hours- ...scientific discovery to powering AI and the technologies people... ...are seeking an experienced software systems engineer to join AMD's Datacenter... ...organization through design reviews, code reviews, troubleshooting... ...drive timely resolution.Evaluate emerging technologies and industry...Suggested
- ...You will be expected to utilize AI coding assistants and LLMs... ...impact architectural and strategic engineering challenges.We are hiring Senior Backend Software Engineers to join our team. Three... ...feedback and supportExperience evaluating trade-offs between design approachesSoftware...SuggestedTemporary workWork at officeLocal area
- ...Number: 200693427Team: Software and ServicesLocation: AustinPosted... .... IS&T is the engine behind everything Apple... ...to take a service from code to running in... ...outsized reach, bringing an AI-first mindset to how the... ...workflows — along with the evaluations and quality controls...Suggested
- ...General Motors is seeking a Senior Software Engineer to support, design, and... ...Engineer will also help introduce AI-enabled capabilities in... ...applications and infrastructure as code using Terraform and similar... ...ability to evaluate, design, and implement deployment...Full timeLocal areaWork from homeRelocation package
- ...hiring a Senior Cloud Security Engineer who builds. Most of your week goes to writing code and infrastructure: Terraform modules... ...hands-on AWS: IAM policy evaluation, Organizations, VPC and private... ...Azure, and limited GCP (Vertex AI)Kubernetes and a microservice architecture...Work at officeRemote workFlexible hours
- ...for a highly motivated Senior Software Engineer who is passionate about... ...software engineering principles, coding expertise, and automation... ...practices, automation, and AI-assisted capabilities to improve... ....Experience in creating evaluation frameworks, validation strategies...Full timeWork at office
- ...Motors’ Advanced Security Engineering team develops advanced... ...intelligence (AI) capabilities to strengthen... ...security findings with code, cloud, and runtime context... ...boundaries. You will evaluate options, exercise sound... ...years of experience in software engineering, cloud engineering...Full timeLocal areaWork from homeWorldwideRelocation package
- ...Number: 200684574Team: Software and ServicesLocation: AustinPosted... .... IS&T is the engine behind everything Apple... ...what you could do here.AI & Data Platforms (AiDP)... ...tuning, and model evaluation, enabling rapid experimentation... ...efficient SQLs.Strong coding and software...
$104k - $187k
...security with the world’s most advanced AI-native platform. We work on large scale... ...with you. About the Role: As our new software engineer in CrowdStrike, you’ll contribute to the... ...team and mentor team members through code reviews, pairing, documentation, design...Permanent employmentWork experience placementWork at officeLocal area2 days per week3 days per week$155k - $233k
...March 2026) Day to Day As a Software Engineer IV at Indeed, you will own... ...Architect new solutions, evaluate trade-offs, and validate ideas... ...Participate in and influence code and design reviews across teams... ...resume for that opening. AI Notice Indeed is committed...Temporary workWork experience placementLocal area$140k - $220k
...world’s most advanced AI-native platform. We work... ...directly with product engineering teams and their leadership... ...— writing production code, re‑architecting critical... ...groups.Continuously re‑evaluate our products to improve... ...community; evangelize software engineering best...Full timeWork experience placementWork at officeLocal area2 days per week3 days per week$116.2k - $229.1k
...027The Work You’ll DoThe Software & ALM Senior Consultant executes... ...-on delivery to build AI-enabled software and systems engineering lifecycle workflows (... ...approvals and auditability.Build evaluation approaches/harnesses to... ...requirements, design, code, test, and release (...Local area- Austin, TXEngineering - Software /Full-time /On-... ...add a Senior Software Engineer to our team. In this role... ...implementation, debugging, code integration, unit testing... ...development activities.Evaluate software requirements,... ...artificial intelligence (AI) tools to support parts...Permanent employmentFull timeContract workWork experience placementWork at officeLocal area
- ...Industry Services team is the engine behind that data — we... ...and energetic Senior Software Engineer to join the... ...will involve investing in AI-driven solutions to... ...efficiency. You will ship code that ensures the flow... ...emerging technologies; evaluate and recommend tools and...Work at officeLocal area
$184k - $287.5k
...unlimited potential of AI to define the next era of... ...excited about open-source software, developer tools, and... ...looking for a Senior Software Engineer to help build the core libraries, evaluation systems, and reusable... ...extending LLM agents, coding agents, tool-use loops,...Full timeRemote work$184k - $292.56k
At Scale, we build the AI systems behind some of... ...On the Public Sector Engineering team, you'll embed directly... ...production, and ship software that customers adopt... ....Strong production coding in Python and TypeScript... ...a fair and thorough evaluation of all applicants.About...Full timeRelocation$224k - $356.5k
...into the unlimited potential of AI to define the next era of... ...NVIDIA is seeking a dedicated Software Engineer, ML Performance to join our team... ...analysis others can reuse.Evaluate automated diagnoses: judge whether... ...by maintainable analysis code, tests or repeatable...Full time$139.3k - $203.6k
...across Cisco's enterprise software portfolio. Our mission... ...experience. Our engineers solve complex distributed... ...through design reviews, code reviews, automated testing... ...customer issues.Evaluate emerging technologies and... ...protect organizations in the AI era - and beyond. We’ve...Full timeTemporary workWork at officeLocal areaRemote workWorldwideFlexible hours$22.1 - $42.5 per hour
...standards for technologies like AI, data centers, automotive and... ...(MACH) division is KLA's software powerhouse, responsible for developing... ...scientists, software engineers, apps engineers and process engineers... ...with IC desgin guides Evaluating and improving machine...Hourly payMinimum wageInternship- ...Number: 200694400Team: Software and ServicesLocation: AustinWeekly... .... IS&T is the engine behind everything Apple... ...to take a service from code to running in... ...outsized reach, bringing an AI-first mindset to how the... ...how you designed it and evaluated its reliabilityStrong fundamentals...
$152k - $241.5k
...collaborate with architecture, software, product, and research... ...Deep Learning Software Engineer to help develop... ...software rapidly assesses new AI-accelerating GPU... ...enable fast, scalable evaluation of AI workloads across... ...learning compiler kernel code generation and computational...Full time- ...implementing secure, reliable AI-powered tools and... ...them into core engineering workflows to improve quality... ...integrity at scale. Define coding and documentation... ...Delivery: Create and maintain software solutions within the... ...management.Develop and evaluate prompts and systems with...Work experience placementWork at officeLocal areaRemote workMonday to ThursdayFlexible hours1 day per week
- Job DescriptionAdvantest is seeking a Software Engineer to join the US AI and Automation Team. In this role, you will help... ...based workflows including source control, code reviews, issue management, and release processes.Evaluate new technologies and development tools to...
- ...We are seeking a Sr. Backend Software Engineer — News & Insights (T3) to... ...initiatives—including our upcoming AI-powered hyperlocal content... ...: Integrate modern AI coding tools into daily development... ...engineering velocity while evaluating generated code for security...Local area
$50 per hour
...DescriptionJoin our team as a Staff AI Engineer focused on ontologies,... ...governance, source traceability, evaluation, version control, and human... ....Review architecture and code for quality, security, supportability... ...developing production software in Python and architecting distributed...Full timeTemporary workWork experience placementCasual workFlexible hours- Job-ID: 19854Category: Software EngineeringLocation: Austin... ...support.The engineer applies strong software... ...through design discussions, code reviews, and mentoring... ...features and services, evaluate technical tradeoffs, and... ...uptime)Experience using AI-assisted development tools...Worldwide
$140.96k - $193.82k
...mitigate risk through actionable AI & data-driven insights. For... ....We are looking for a Senior Software Engineer to help build the next... ...from architectural design and code reviews to observability, latency... ...and leverage operational data, evaluation frameworks, and customer...Full timeWork experience placementWork at officeLocal area- About ZLayt AIZLayt AI is building the next generation of AI-powered software for the legal industry.Our... ...looking for exceptional engineers who want to build... ...maintainable, well-documented code.2. AI-Native... ...and automation features.Evaluate new AI technologies and...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Software Engineer - AI Code Evaluation. Be the first to apply!
- senior software design engineer Austin, TX
- software web developer Austin, TX
- senior software engineer Austin, TX
- labview software developer Austin, TX
- software developer fintech Austin, TX
- software support engineer Austin, TX
- security software engineer Austin, TX
- software developer night shift Austin, TX
- integration software engineer Austin, TX
- scientific software engineer Austin, TX




