Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Software Engineer - AI Code Evaluation

$75 - $105 per hour

Braintrust

About the Opportunity

Review Python code, tests, and execution logs from AI coding tasks to determine why a solution failed and document the evidence behind your conclusion.

You’ll work with a leading AI research organization to evaluate how coding agents perform on software engineering tasks. A failing test might point to a bug in the agent’s code, an overly restrictive assertion, an unclear requirement, or a problem with the test environment. Your job is to trace what happened and explain whether it affected the result.

What You’ll Do

  • Read unfamiliar code and trace an AI coding agent’s changes through the task requirements, test suite, and run logs.
  • Reproduce failures and identify what caused them. Check whether the implementation, test assertion, task specification, or environment is responsible.
  • Diagnose test harness and environment failures using Docker configuration, dependency information, and execution logs.
  • Determine whether the issue changed the task’s pass/fail result. Write a concise finding that cites the relevant code, test, requirement, or log and explains how you verified it.
  • Join daily discussions with research scientists to compare findings and apply consistent evaluation criteria.

A typical investigation takes 30–60 minutes .

What We’re Looking For

  • 2+ years of professional experience writing, reviewing, and debugging software , or a completed STEM master’s degree or PhD with intensive programming and computational research experience.
  • Hands-on experience reading and debugging Python in unfamiliar or complex codebases. You can walk through a real failure you investigated, explain what evidence isolated the cause, and describe how you verified the fix.
  • Ability to read at least one additional language, such as JavaScript or TypeScript, Go, C++, or Rust .
  • Practical experience with Git, Docker, and software build or test environments .
  • Experience with a test framework such as pytest, unittest, Jest, or Mocha . You can read test assertions and assess whether they match the documented behavior.
  • Clear written and spoken English for explaining technical findings and the evidence behind them.

Nice to Have

  • Experience evaluating LLMs or coding agents, including reviewing their code changes, test results, or execution traces.
  • Familiarity with coding benchmarks such as SWE-bench or CodeContests.
  • Experience with AI reliability evaluation or collaborating with ML researchers.

Engagement Details

  • Location — Remote, United States only
  • Expected commitment — 35 hours per week
  • Start — Immediate
  • Compensation — $75–$105 USD per hour, depending on relevant skills and experience

Interview Process

Shortlisted candidates will need to complete a 15-minute AI interview.

Vacancy posted 3 hours ago
Similar jobs that could be interesting for youBased on the Software Engineer - AI Code Evaluation in Austin, TX vacancy
  • $150k - $270k

     ...with the world’s most advanced AI-native platform. We work on...  ...CrowdStrike is seeking a Senior Software Engineer to help design and build the...  ...is an opportunity to write code that operates at a scale few...  ...globally Continuously re-evaluate architecture, performance, and... 
    Suggested
    Hourly pay
    Permanent employment
    Full time
    Work experience placement
    Work at office
    Local area
    Worldwide
    2 days per week
    3 days per week
    Austin, TX
    19 days ago
  • $143.5k - $212.85k

     ...spanning all phases of the Software Development Lifecycle (...  ...issues, guiding junior engineers, operating with little...  ...key contributor to our AI-powered tooling...  ...assisted infrastructure code generation, and autonomous...  ...engineering workflows, including evaluation, safety, and... 
    Suggested
    Full time
    Work at office
    Local area
    Immediate start
    Flexible hours

    PayPal

    Austin, TX
    4 days ago
  •  ...scientific discovery to powering AI and the technologies people...  ...are seeking an experienced software systems engineer to join AMD's Datacenter...  ...organization through design reviews, code reviews, troubleshooting...  ...drive timely resolution.Evaluate emerging technologies and industry... 
    Suggested

    AMD

    Austin, TX
    1 day ago
  •  ...You will be expected to utilize AI coding assistants and LLMs...  ...impact architectural and strategic engineering challenges.We are hiring Senior Backend Software Engineers to join our team. Three...  ...feedback and supportExperience evaluating trade-offs between design approachesSoftware... 
    Suggested
    Temporary work
    Work at office
    Local area

    Realtor.com

    Austin, TX
    3 days ago
  •  ...Number: 200693427Team: Software and ServicesLocation: AustinPosted...  .... IS&T is the engine behind everything Apple...  ...to take a service from code to running in...  ...outsized reach, bringing an AI-first mindset to how the...  ...workflows — along with the evaluations and quality controls... 
    Suggested

    Apple

    Austin, TX
    1 day ago
  •  ...General Motors is seeking a Senior Software Engineer to support, design, and...  ...Engineer will also help introduce AI-enabled capabilities in...  ...applications and infrastructure as code using Terraform and similar...  ...ability to evaluate, design, and implement deployment... 
    Full time
    Local area
    Work from home
    Relocation package

    General Motors

    Austin, TX
    3 days ago
  •  ...hiring a Senior Cloud Security Engineer who builds. Most of your week goes to writing code and infrastructure: Terraform modules...  ...hands-on AWS: IAM policy evaluation, Organizations, VPC and private...  ...Azure, and limited GCP (Vertex AI)Kubernetes and a microservice architecture... 
    Work at office
    Remote work
    Flexible hours

    Qualia

    Austin, TX
    18 hours ago
  •  ...for a highly motivated Senior Software Engineer who is passionate about...  ...software engineering principles, coding expertise, and automation...  ...practices, automation, and AI-assisted capabilities to improve...  ....Experience in creating evaluation frameworks, validation strategies... 
    Full time
    Work at office

    The Charles Schwab Corporation

    Austin, TX
    3 days ago
  •  ...Motors’ Advanced Security Engineering team develops advanced...  ...intelligence (AI) capabilities to strengthen...  ...security findings with code, cloud, and runtime context...  ...boundaries. You will evaluate options, exercise sound...  ...years of experience in software engineering, cloud engineering... 
    Full time
    Local area
    Work from home
    Worldwide
    Relocation package

    General Motors

    Austin, TX
    4 days ago
  •  ...Number: 200684574Team: Software and ServicesLocation: AustinPosted...  .... IS&T is the engine behind everything Apple...  ...what you could do here.AI & Data Platforms (AiDP)...  ...tuning, and model evaluation, enabling rapid experimentation...  ...efficient SQLs.Strong coding and software... 

    Apple

    Austin, TX
    3 days ago
  • $104k - $187k

     ...security with the world’s most advanced AI-native platform. We work on large scale...  ...with you. About the Role: As our new software engineer in CrowdStrike, you’ll contribute to the...  ...team and mentor team members through code reviews, pairing, documentation, design... 
    Permanent employment
    Work experience placement
    Work at office
    Local area
    2 days per week
    3 days per week
    Austin, TX
    a month ago
  • $155k - $233k

     ...March 2026) Day to Day As a Software Engineer IV at Indeed, you will own...  ...Architect new solutions, evaluate trade-offs, and validate ideas...  ...Participate in and influence code and design reviews across teams...  ...resume for that opening. AI Notice Indeed is committed... 
    Temporary work
    Work experience placement
    Local area

    Indeed

    Austin, TX
    4 days ago
  • $140k - $220k

     ...world’s most advanced AI-native platform. We work...  ...directly with product engineering teams and their leadership...  ...— writing production code, re‑architecting critical...  ...groups.Continuously re‑evaluate our products to improve...  ...community; evangelize software engineering best... 
    Full time
    Work experience placement
    Work at office
    Local area
    2 days per week
    3 days per week

    CrowdStrike

    Austin, TX
    1 day ago
  • $116.2k - $229.1k

     ...027The Work You’ll DoThe Software & ALM Senior Consultant executes...  ...-on delivery to build AI-enabled software and systems engineering lifecycle workflows (...  ...approvals and auditability.Build evaluation approaches/harnesses to...  ...requirements, design, code, test, and release (... 
    Local area

    Deloitte

    Austin, TX
    2 days ago
  • Austin, TXEngineering - Software /Full-time /On-...  ...add a Senior Software Engineer to our team. In this role...  ...implementation, debugging, code integration, unit testing...  ...development activities.Evaluate software requirements,...  ...artificial intelligence (AI) tools to support parts... 
    Permanent employment
    Full time
    Contract work
    Work experience placement
    Work at office
    Local area

    CesiumAstro

    Austin, TX
    2 days ago
  •  ...Industry Services team is the engine behind that data — we...  ...and energetic Senior Software Engineer to join the...  ...will involve investing in AI-driven solutions to...  ...efficiency. You will ship code that ensures the flow...  ...emerging technologies; evaluate and recommend tools and... 
    Work at office
    Local area

    Realtor.com

    Austin, TX
    1 day ago
  • $184k - $287.5k

     ...unlimited potential of AI to define the next era of...  ...excited about open-source software, developer tools, and...  ...looking for a Senior Software Engineer to help build the core libraries, evaluation systems, and reusable...  ...extending LLM agents, coding agents, tool-use loops,... 
    Full time
    Remote work

    Nvidia

    Austin, TX
    1 day ago
  • $184k - $292.56k

    At Scale, we build the AI systems behind some of...  ...On the Public Sector Engineering team, you'll embed directly...  ...production, and ship software that customers adopt...  ....Strong production coding in Python and TypeScript...  ...a fair and thorough evaluation of all applicants.About... 
    Full time
    Relocation

    Scale AI

    Austin, TX
    5 days ago
  • $224k - $356.5k

     ...into the unlimited potential of AI to define the next era of...  ...NVIDIA is seeking a dedicated Software Engineer, ML Performance to join our team...  ...analysis others can reuse.Evaluate automated diagnoses: judge whether...  ...by maintainable analysis code, tests or repeatable... 
    Full time

    Nvidia

    Austin, TX
    3 days ago
  • $139.3k - $203.6k

     ...across Cisco's enterprise software portfolio. Our mission...  ...experience. Our engineers solve complex distributed...  ...through design reviews, code reviews, automated testing...  ...customer issues.Evaluate emerging technologies and...  ...protect organizations in the AI era - and beyond. We’ve... 
    Full time
    Temporary work
    Work at office
    Local area
    Remote work
    Worldwide
    Flexible hours

    CISCO Systems

    Austin, TX
    1 day ago
  • $22.1 - $42.5 per hour

     ...standards for technologies like AI, data centers, automotive and...  ...(MACH) division is KLA's software powerhouse, responsible for developing...  ...scientists, software engineers, apps engineers and process engineers...  ...with IC desgin guides Evaluating and improving machine... 
    Hourly pay
    Minimum wage
    Internship

    KLA

    Austin, TX
    1 day ago
  •  ...Number: 200694400Team: Software and ServicesLocation: AustinWeekly...  .... IS&T is the engine behind everything Apple...  ...to take a service from code to running in...  ...outsized reach, bringing an AI-first mindset to how the...  ...how you designed it and evaluated its reliabilityStrong fundamentals... 

    Apple

    Austin, TX
    5 days ago
  • $152k - $241.5k

     ...collaborate with architecture, software, product, and research...  ...Deep Learning Software Engineer to help develop...  ...software rapidly assesses new AI-accelerating GPU...  ...enable fast, scalable evaluation of AI workloads across...  ...learning compiler kernel code generation and computational... 
    Full time

    Nvidia

    Austin, TX
    1 day ago
  •  ...implementing secure, reliable AI-powered tools and...  ...them into core engineering workflows to improve quality...  ...integrity at scale. Define coding and documentation...  ...Delivery: Create and maintain software solutions within the...  ...management.Develop and evaluate prompts and systems with... 
    Work experience placement
    Work at office
    Local area
    Remote work
    Monday to Thursday
    Flexible hours
    1 day per week

    Roku

    Austin, TX
    5 days ago
  • Job DescriptionAdvantest is seeking a Software Engineer to join the US AI and Automation Team. In this role, you will help...  ...based workflows including source control, code reviews, issue management, and release processes.Evaluate new technologies and development tools to... 

    Advantest

    Austin, TX
    1 day ago
  •  ...We are seeking a Sr. Backend Software Engineer — News & Insights (T3) to...  ...initiatives—including our upcoming AI-powered hyperlocal content...  ...: Integrate modern AI coding tools into daily development...  ...engineering velocity while evaluating generated code for security... 
    Local area

    Realtor.com

    Austin, TX
    1 day ago
  • $50 per hour

     ...DescriptionJoin our team as a Staff AI Engineer focused on ontologies,...  ...governance, source traceability, evaluation, version control, and human...  ....Review architecture and code for quality, security, supportability...  ...developing production software in Python and architecting distributed... 
    Full time
    Temporary work
    Work experience placement
    Casual work
    Flexible hours

    Lockheed Martin

    Austin, TX
    4 days ago
  • Job-ID: 19854Category: Software EngineeringLocation: Austin...  ...support.The engineer applies strong software...  ...through design discussions, code reviews, and mentoring...  ...features and services, evaluate technical tradeoffs, and...  ...uptime)Experience using AI-assisted development tools... 
    Worldwide

    Resideo Technologies

    Austin, TX
    2 days ago
  • $140.96k - $193.82k

     ...mitigate risk through actionable AI & data-driven insights. For...  ....We are looking for a Senior Software Engineer to help build the next...  ...from architectural design and code reviews to observability, latency...  ...and leverage operational data, evaluation frameworks, and customer... 
    Full time
    Work experience placement
    Work at office
    Local area

    Procore Technologies

    Austin, TX
    18 hours ago
  • About ZLayt AIZLayt AI is building the next generation of AI-powered software for the legal industry.Our...  ...looking for exceptional engineers who want to build...  ...maintainable, well-documented code.2. AI-Native...  ...and automation features.Evaluate new AI technologies and... 

    Zinda Law Group

    Austin, TX
    1 day ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Software Engineer - AI Code Evaluation. Be the first to apply!