Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Software Engineer - AI Code Evaluation

$75 - $105 per hour

Braintrust

About the Opportunity

Review Python code, tests, and execution logs from AI coding tasks to determine why a solution failed and document the evidence behind your conclusion.

You’ll work with a leading AI research organization to evaluate how coding agents perform on software engineering tasks. A failing test might point to a bug in the agent’s code, an overly restrictive assertion, an unclear requirement, or a problem with the test environment. Your job is to trace what happened and explain whether it affected the result.

What You’ll Do

  • Read unfamiliar code and trace an AI coding agent’s changes through the task requirements, test suite, and run logs.
  • Reproduce failures and identify what caused them. Check whether the implementation, test assertion, task specification, or environment is responsible.
  • Diagnose test harness and environment failures using Docker configuration, dependency information, and execution logs.
  • Determine whether the issue changed the task’s pass/fail result. Write a concise finding that cites the relevant code, test, requirement, or log and explains how you verified it.
  • Join daily discussions with research scientists to compare findings and apply consistent evaluation criteria.

A typical investigation takes 30–60 minutes .

What We’re Looking For

  • 2+ years of professional experience writing, reviewing, and debugging software , or a completed STEM master’s degree or PhD with intensive programming and computational research experience.
  • Hands-on experience reading and debugging Python in unfamiliar or complex codebases. You can walk through a real failure you investigated, explain what evidence isolated the cause, and describe how you verified the fix.
  • Ability to read at least one additional language, such as JavaScript or TypeScript, Go, C++, or Rust .
  • Practical experience with Git, Docker, and software build or test environments .
  • Experience with a test framework such as pytest, unittest, Jest, or Mocha . You can read test assertions and assess whether they match the documented behavior.
  • Clear written and spoken English for explaining technical findings and the evidence behind them.

Nice to Have

  • Experience evaluating LLMs or coding agents, including reviewing their code changes, test results, or execution traces.
  • Familiarity with coding benchmarks such as SWE-bench or CodeContests.
  • Experience with AI reliability evaluation or collaborating with ML researchers.

Engagement Details

  • Location — Remote, United States only
  • Expected commitment — 35 hours per week
  • Start — Immediate
  • Compensation — $75–$105 USD per hour, depending on relevant skills and experience

Interview Process

Shortlisted candidates will need to complete a 15-minute AI interview.

Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the Software Engineer - AI Code Evaluation in New York, NY vacancy
  • $90 per hour

     ...Pay: Up to $90+/hr Experienced software engineers review and score AI-written code for frontier AI research teams. Remote and flexible, up to 40 hours a week. Pay is up to $90+ per hour for the most experienced engineers. Requirements: several years of professional... 
    Suggested
    Hourly pay
    Temporary work
    For contractors
    Live in
    Remote work
    Visa sponsorship
    Flexible hours

    rustLabs

    New York, NY
    21 days ago
  • $174k - $252k

     ...infra blocks, such as streaming evaluation, autorater pipelines, and...  ....5 years of experience with coding in Python.3 years of...  ...experience working with Generative AI agents.2 years of experience...  ...in Python and C++.Google's software engineers develop the next-generation... 
    Suggested

    Google

    New York, NY
    4 days ago
  • $175k - $315k

     ...world’s most advanced AI-native platform. We work...  ...CrowdStrike Cloud Detection Engine (CDE) team is building...  ...engine that evaluates trillions of events a day...  ...paradigms to effective peer code reviews and resilient architecture...  ...produce high-quality software that is unit tested,... 
    Suggested
    Permanent employment
    Work experience placement
    Work at office
    Local area
    2 days per week
    3 days per week
    New York, NY
    a month ago
  • $136k - $245k

     ...with the world’s most advanced AI-native platform. We work on...  ...development process, so vulnerable code is identified as it is written...  ...secure applications. Our engineers work with AI coding agents...  ...through to resolution. Keep re-evaluating our architecture, test... 
    Suggested
    Permanent employment
    Work experience placement
    Work at office
    Local area
    3 days per week
    New York, NY
    5 days ago
  • $197.3k - $225.1k

    AI Engineer 4 (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) At Capital One, we are creating responsible and reliable AI...  ...test, deploy, and support AI software components including foundation...  ...Francisco, California Police Code Article 49, Sections 4901-4920... 
    Suggested
    Full time
    Part time
    Local area

    Capital One

    New York, NY
    1 day ago
  • $229.9k - $262.4k

    AI Engineer 5 (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) At Capital One, we are creating responsible and reliable AI...  ...test, deploy, and support AI software components including foundation...  ...Francisco, California Police Code Article 49, Sections 4901-4920... 
    Full time
    Part time
    Local area

    Capital One

    New York, NY
    1 day ago
  • $229.9k - $262.4k

    Sr. Manager, AI Engineer (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) At Capital One, we are creating responsible and...  ..., deployment, and support AI software components including foundation...  ...Francisco, California Police Code Article 49, Sections 4901-4920... 
    Full time
    Part time
    Local area

    Capital One

    New York, NY
    1 day ago
  • $164k - $295k

     ...world’s most advanced AI-native platform. We work...  ...directly with product engineering teams and their leadership...  ...— writing production code, re‑architecting critical...  .... Continuously re‑evaluate our products to improve...  ...community; evangelize software engineering best practices... 
    Permanent employment
    Work experience placement
    Work at office
    Local area
    2 days per week
    3 days per week
    New York, NY
    29 days ago
  • $159.8k - $235k

    About the TeamThe Code Quality team sits within the Developer Platform...  ...is to make sure the signals engineers rely on before shipping — test...  ...the RoleWe're looking for Software Engineers to help build and maintain...  ..., and rolloutsExplore how AI agents and tools can automate... 
    Hourly pay
    Work at office
    Local area
    Remote work
    Flexible hours

    Doordash

    New York, NY
    1 day ago
  •  ...Financial Services & InsuranceAI Evaluation Engineer PRIMARY PURPOSE: The AI Evaluation Engineer designs, builds...  ...preferred. ~ Experience ~4+ years in software, ML, or data engineering, with at...  ...(testing, version control, CI/CD, code review). ~ Practical experience... 
    Remote work

    Sedgwick

    New York, NY
    5 days ago
  • $121.79k - $166.95k

     ...have an exciting opportunity to join our team as a Lead Software Engineer - AI Coding. As a Lead Software Engineer in AI Coding, you will be responsible...  ...software products based on published maturity model Evaluate tools and technologies and provide analysis and... 
    Full time
    Contract work
    Flexible hours

    NYU Langone Medical Center

    New York, NY
    4 days ago
  • $220k - $300k

     ...organizations use to build their own clinical AI agents. Our mission is to make...  .... About this role As a Software Engineer, Voice Evaluation at Concurrence, you'll build the system...  ...through to a verified fix You use coding agents in your work and independently... 
    Full time
    Flexible hours

    Concurrence

    New York, NY
    5 days ago
  •  ...Senior Software Engineer - Job Description Location: Remote (Anywhere in the USA) We are...  ...Software Engineer who thrives on writing code and pushing projects to production...  ...Best Practice Evolution: Research, evaluate, and implement modern software engineering... 
    Full time
    Contract work
    Immediate start
    Remote work

    HRC Global Services

    New York, NY
    1 day ago
  •  ...Company Cassidy is a no-code platform that allows...  ...businesses to build custom AI assistants that...  ...faster without heavy engineering overhead. About the...  ...years of professional software development experience...  ...AI : Familiarity with evaluating AI models or integrating... 

    Cassidy

    New York, NY
    17 days ago
  • $164k - $295k

     ...modern security with the world’s most advanced AI-native platform. We work on large scale...  ...'ll harness cutting-edge technologies to engineer robust backend services that seamlessly...  ...experience including testing strategies, code reviews, and resilient architecture design... 
    Permanent employment
    Work experience placement
    Work at office
    Local area
    Worldwide
    2 days per week
    3 days per week
    New York, NY
    a month ago
  • $10,500 per month

     ...builds the world's leading software for data-driven decisions...  ...The Role As a Software Engineer Intern on Sovereign AI Infrastructure, you will help...  ...identity or permissions, evaluations, and the core workflows...  ...looking for interns with strong coding fundamentals and... 
    Full time
    Work experience placement
    Internship
    Work at office
    Remote work
    Work from home

    Palantir

    New York, NY
    1 day ago
  • $125k - $225k

     ...modern security with the world’s most advanced AI-native platform. We work on large scale...  ...Relentlessly pursue quality by following engineering best practices including: Appropriate testing paradigms, effective peer code reviews, logging, monitoring, and resilient... 
    Permanent employment
    Work experience placement
    Work at office
    Local area
    3 days per week
    New York, NY
    a month ago
  •  ...Senior Software Engineer, Agent Platform New York, NY A recently-minted...  ...investors — is building the AI infrastructure powering human...  ...autonomous agents get built, evaluated, and trusted in production, inventing...  ...assisted development (Claude Code, Cursor, or similar) as a... 
    Full time

    Integra Tech Advisors

    New York, NY
    3 days ago
  • $157k - $235k

     ...digital services.Snap Engineering teams build fun and technically...  ...product requirements, evaluate trade-offs, and deliver...  ...cost managementUtilize AI tools and high velocity...  ...rigorous standards for code correctness, security,...  ...of post-Bachelor’s software development experience;... 
    Full time
    Live in
    Work at office
    Local area

    Snap

    New York, NY
    2 days ago
  • $180k - $260k

     ...modern platforms and AI, you’ll do the most meaningful...  ...is the innovation and engineering hub behind Epiq’s next...  ..., retrieval, and evaluation depend on.Instrument...  ...-call rotation.Use AI coding agents throughout development...  ...delivering working software.Required... 
    Full time
    Work experience placement
    Local area

    Epiq Systems

    New York, NY
    2 days ago
  •  ...Senior Software Engineer | AI-First Development New York City – Financial District 4 Days/Week...  ...Windsurf to accelerate development, improve code quality, automate workflows, and solve...  ...productivity. Continuously evaluate emerging AI development tools and techniques... 
    Relocation

    eNGINE

    New York, NY
    13 days ago
  • $124.9k - $167.4k

     ...global organization of engineers, product developers, designers...  ...Job Summary As a Software Engineer II, you will...  ...and build intelligent, AI driven systems that...  ...should be measured and evaluated for reliability and compliance...  ...(e.g., Cursor, Claude Code) to accelerate... 

    Disney Entertainment and ESPN Product & Technology Careers

    New York, NY
    9 days ago
  •  ...Full Stack Engineer NYC MUST BE ELIGIBLE FOR SECURITY CLEARANCE...  ...Key Responsibilities Evaluate, refine, and improve AI-powered search and review...  ..., and maintainable code while contributing to a collaborative...  ...~3+ years of professional software engineering experience.... 
    Full time

    Green Key Resources

    New York, NY
    1 day ago
  • $164k - $295k

     ...modern security with the world’s most advanced AI-native platform. We work on large scale...  ...We're looking for a Senior UI Platform Engineer to lead the design and build of the shared...  ...scaffold, build, integrate and ship their code across the organisation. UI Platform owns... 
    Permanent employment
    Contract work
    Work experience placement
    Work at office
    Local area
    Remote work
    Shift work
    New York, NY
    10 days ago
  •  ...distress. They are seeking a full-time Software Engineer (Data), AI Platform to work fully onsite out of...  ...generation, vector search, and evaluation frameworks • Proficiency with AI-assisted...  ...development tooling (e.g., Claude Code, Cursor) as a primary development... 
    Full time
    Work at office

    Atlantic Group

    New York, NY
    15 days ago
  •  ...Senior Software Engineer – AI | Forward Deployed Engineering Location: Hybrid, with required office...  ...usage, and system health. Develop evaluation pipelines and benchmarks to assess...  ..., Azure, or GCP and infrastructure as code tools such as Terraform or CDK. ~ Experience... 
    Full time
    Contract work
    Work at office

    kadence

    New York, NY
    1 day ago
  • $176k - $191k

     ...TeamDeveloper Experience & AI Velocity is a new team...  ...experience for Wonder Engineering — 400+ engineers...  ...and controlled costs.AI coding tools are now a...  ...testing, and debugging software at Wonder, but adoption...  ...with an objective lens — evaluating people for their potential... 
    Full time
    Temporary work
    Work at office
    Flexible hours
    3 days per week

    GrubHub

    New York, NY
    11 hours ago
  • $117.7k - $218.6k

    Senior Software Engineer, Search Platform Our vision We are building the next...  ...platform. Job DescriptionThe AI Enabled Software Engineering...  ...retrieval components so offline evaluation and LLM-as-judge grading can...  ...) and infrastructure as code Operational ownership: on-call... 
    Full time
    Work at office
    Local area
    Flexible hours

    Thomson Reuters

    New York, NY
    3 hours ago
  • $141k

     ...ecosystem of devices and cloud software. Like our products, we...  ...infrastructure with an AI intelligence layer—...  ...experienced platform engineer to join Prepared's...  ...knowledge of infrastructure-as-code tools like Terraform,...  ...from applicants to evaluate candidates for employment... 
    Work experience placement
    Work at office

    Axon

    New York, NY
    4 days ago
  • $148.5k - $223.9k

     ...SalesforceSalesforce is the #1 AI CRM, where humans with agents...  ...’re looking for a Senior Software Engineer to take important, poorly defined...  ...into what we deliver. Evaluate design choices against enterprise...  ...effectiveness of the team through code reviews, clear decisions, and... 
    Full time

    Salesforce

    New York, NY
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Software Engineer - AI Code Evaluation. Be the first to apply!