Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Software Engineer - AI Code Evaluation & Benchmarking (US candidates only) [Remote]

AuraOne Human Data

Remote
  • Remote job

Software Engineer - AI Code Evaluation & Benchmarking (US candidates only) is a remote engineering review track for evaluating production code, debugging traces, and developer-facing AI outputs against real-world correctness standards. Reviewers reproduce failures, write the unit test the model should have written, and explain the fix so the modeling team can target the gap.

Why this role matters

Engineering model quality lives or dies on whether the generated code actually compiles, passes tests, and handles edge cases. AuraOne pairs experienced engineers with the modeling team to grade outputs the way a code reviewer would.

Responsibilities

  • Run and reproduce candidate code outputs in a sandboxed environment for Software Engineer - AI Code Evaluation & Benchmarking (US candidates only) assignments.
  • Grade software engineer ai code evaluation benchmarking engineering review solutions for correctness, style, and edge-case handling.
  • Write minimal failing tests that demonstrate the bug a model output missed.
  • Compare paired solutions and rank them with a written rationale tied to the rubric.
  • Tag failure modes (compile error, runtime crash, off-by-one, security issue) with severity scores.
  • Document recurring code-generation failures so the modeling team can target them.
  • Calibrate against gold-standard reviews to keep inter-rater agreement above target.

Qualifications

  • Strong day-job engineering experience — you can read, run, and debug unfamiliar code for Software Engineer - AI Code Evaluation & Benchmarking (US candidates only) work.
  • Comfort writing concise unit tests that capture a single failure mode.
  • Familiarity with a testing framework. pytest, Jest, or JUnit. Go test, RSpec, or whatever you use.
  • Clear written reasoning — your review note has to convince another senior engineer.
  • Reliable async availability for at least 10 hours per week.
  • Prior code-review or technical-interview-grading experience is a plus.

Example tasks

  • Reproduce a generated engineering solution to a coding task, run the test suite, and grade it.
  • Write the smallest failing test that demonstrates a model's edge-case bug.
  • Compare two paired solutions and rank them with a written rationale tied to the rubric.
  • Triage a security issue surfaced by a model and document the patch the model should have produced.

Nice to have

  • Open-source contributions or a public portfolio that demonstrates production-quality code.
  • Experience with the target language's standard tooling, linters, and idiomatic style guides.
  • Familiarity with security-review checklists (OWASP, CWE) and AppSec patterns.

Skills

  • Code review
  • Debugging
  • Unit testing
  • Software engineering judgment
  • Software Engineer AI Code Evaluation Benchmarking engineering review
  • Python
  • Git
  • Docker
  • Legal & Law Services

Work model

Remote — US-eligible. Remote · Independent specialist contractor. Employment type: CONTRACTOR. Applicants must be authorized to work from US.

Compensation

Hourly rate confirmed after the interview process.

Application process

Apply through AuraOne's specialist intake for role-specific routing and review. Final project scope, schedule, and contractor terms are confirmed before placement.

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Software Engineer - AI Code Evaluation & Benchmarking (US candidates only) [Remote] in Remote vacancy
  • $75 - $105 per hour

     ...Opportunity Review Python code, tests, and execution logs from AI coding tasks to...  ...organization to evaluate how coding agents perform on software engineering tasks. A failing...  ...with coding benchmarks such as SWE-bench...  ...Process Shortlisted candidates will need to complete... 
    Suggested
    Hourly pay
    Immediate start
    Remote work

    Braintrust

    Austin, TX
    17 hours ago
  • $197.3k - $225.1k

     ...Job Description AI Engineer 4 (AI Foundations: Benchmarking, Evaluation, and...  ...learning — position us to be at the forefront...  ...deploy, and support AI software components including...  ...translation The Ideal Candidate:  You love to...  ...California Police Code Article 49,... 
    Suggested
    Full time
    Part time
    Local area

    Capital One

    McLean, VA
    19 days ago
  • $90 per hour

     ...Pay: Up to $90+/hr Experienced software engineers review and score AI-written code for frontier AI research teams. Remote and flexible, up to 40 hours...  ...professional software engineering experience; you live in the US or Canada and can work as an independent contractor... 
    Suggested
    Hourly pay
    Temporary work
    For contractors
    Live in
    Remote work
    Visa sponsorship
    Flexible hours

    rustLabs

    New York, NY
    22 days ago
  •  ...About Us Based in San Francisco...  ...for frontier AI labs and a trusted...  ...specialize in software engineering, logical...  ...Software Engineering evaluator, you will...  ...for training, benchmarking, and advancing...  ...includes curating code examples,...  ...Location: Candidates must be based... 
    Suggested
    Contract work
    For contractors
    Flexible hours

    Turing

    Remote
    13 days ago
  •  ...About Us Based in San Francisco...  ...for frontier AI labs and a...  ...specialize in software engineering, logical reasoning...  ...Engineering evaluator, you will create...  ...for training, benchmarking, and advancing...  ...curating code examples, providing...  ...) Location: candidates must be based... 
    Suggested
    Contract work
    For contractors
    Flexible hours

    Turing

    Remote
    13 days ago
  • $139.2k - $235.2k

     ...more secure software faster. The...  ...: we embrace AI as a core productivity...  ...future with us as we build...  ...Backend Engineer on GitLab's Security...  ...Factory: Code Security team...  ...develops, evaluates, and ships those...  ...against benchmark applications...  ...interest from candidates with varying... 
    Full time
    Live in
    Remote work
    Flexible hours

    GitLab

    Remote
    16 days ago
  • $80 - $100 per hour

     ...Senior Python Developer (AI Evaluation & Benchmarking) An enterprise client...  ...evaluating AI-generated code, and helping improve the...  ...opportunity for experienced software engineers who enjoy solving complex...  ...the client. Qualified candidates will receive an Upwork contract... 
    Hourly pay
    Weekly pay
    Contract work
    Remote work
    10 hours per week

    Lifted, an Upwork Company™

    United States
    3 days ago
  •  ...About Us Our mission is to raise...  ...was founded by engineers and researchers...  ...next generation of AI. Our platform powers...  ...rigorous evaluations that go beyond benchmarks. We've run a profitable...  ...Engineer, Coding Evaluation & Training...  ...intersection of software engineering and... 
    Full time

    Surge AI

    Remote
    13 days ago
  •  ...clients across the US. What makes us different...  ...out. Title: Software Engineer - AI Evaluation & Automation...  ...harnesses, automate benchmark runs, and help make...  ...experience using AI coding tools and agentic harnesses...  ...zone. Not accepting candidates with more than a two... 
    For contractors
    Remote work
    Flexible hours

    HonorVet Technologies

    San Diego, CA
    26 days ago
  • $10k

     ...LabsRamp Labs is Ramp's AI research group....  ...runs through a benchmark. Which model to...  ...review, invoice coding, month-end close,...  ...decisionsStrong Python and software engineering; your eval code...  ...benchmark or evaluation research (NeurIPS...  ...and level at which the candidate is hired.
    Full time
    Work at office
    Home office
    Flexible hours

    Ramp

    New York, NY
    3 days ago
  • $144.7k - $221.4k

     ...unprecedented scale. Join us to help deliver...  ...The Evaluation team builds and...  ...clear feedback for engineering and leadership,...  ...driving software performance at...  ...system design, code reviews, testing...  ...Experience leveraging AI-assisted...  ...the successful candidate is expected to... 
    Full time
    Work at office
    Local area
    Work from home
    Relocation
    Relocation package
    Flexible hours

    General Motors

    Sunnyvale, TX
    3 days ago
  • $140k - $175k

    Senior Software Engineer (Hybrid - US) Boston, Massachusetts, United States...  ...Software Engineer evaluates technical requirements...  ...level engineers through code reviews, technical...  ...opportunities. We base our candidate selection on one's...  ...intelligence (AI) tools (including Generative... 
    Work at office
    Remote work

    Energy Solutions USA, Inc

    Chicago, IL
    3 days ago
  •  ...your career with us, experience our caring...  ...& Insurance AI Evaluation Engineer PRIMARY PURPOSE:...  ...to U.S. based candidates. ESSENTIAL FUNCTIONS...  ...evaluation (benchmark and golden...  ...Experience 4+ years in software, ML, or data...  ...control, CI/CD, code review). Practical... 
    Full time
    Local area
    Remote work

    Sedgwick

    Remote
    19 days ago
  •  ...Epoch AI is looking for a Software Engineer, Cybersecurity Benchmarks who will help us measure how capable frontier AI models...  ..., securing our evaluation environments against...  ...releases quickly. An ideal candidate for this role might...  ...familiarity with AI coding harnesses. Your work... 
    Full time
    Remote work
    Flexible hours

    Epoch Ai

    Remote
    12 days ago
  • $250k - $350k

     ...is the Work AI platform that...  ...’t writing code anymore. It’...  ...Manager of AI Evaluations, you will...  ...scientists and engineers and build...  ...experience in software engineering,...  ...datasets, benchmark creation, deterministic...  ...for Job Candidates and...  ...as required. US applicants and... 
    Work at office
    Home office
    Flexible hours

    Glean Work

    Mountain View, CA
    13 days ago
  • Software Engineer III Full Time - Remote Cincinnati...  ...Innovate with Benchmark Gensuite as a Senior...  ...inclusion. Join us and help make...  ...clients. The ideal candidate will perform coding, debugging,...  ...build processes AI & Emerging Technology...  ...feedback Evaluate and integrate AI... 
    Full time
    Immediate start
    Remote work
    Worldwide
    Flexible hours

    Benchmark Gensuite®

    New York, NY
    3 days ago
  • $107k - $160k

     ...Netherlands. Visit us at We are...  ...care about our candidates, employees and...  ...passionate Senior Cloud Engineer to join our...  ...-as-code best practices...  ...: Partner with software engineers and other...  ...Experience using AI-powered coding...  ...to critically evaluate and validate generated... 
    Full time
    Remote work
    Work from home
    Worldwide
    Home office

    Mitek Systems

    Remote
    4 days ago
  • $98.61k - $167.64k

     ...experienced and driven Senior-Level Software Engineer to lead and oversee mission-...  ...software solutions. Conduct code reviews, unit and...  ...****@*****.*** for assistance.Candidate AI Usage PolicyAt ICF, we are...  ...are instructed to contact us in advance at... 
    Full time
    Contract work
    Work experience placement
    Interim role
    Work at office
    Remote work

    ICF

    Reston, VA
    3 days ago
  • $87.8k - $209.5k

     ...Clinical Applications Engineering Transform...  ...skilled Principal Software Engineer to join our...  ...Engineering Productivity & AI Enablement...  ...tools, including coding assistants and prompt...  ...Role The ideal candidate is first and foremost...  ...locations only US: Hiring Range in... 
    Full time
    Temporary work
    Worldwide
    Flexible hours

    Oracle

    Remote
    16 days ago
  • $99.6k - $174k

    Senior Full Stack Engineer, AI Platform & AgentsBuild...  ...scale.-Location:: US/Canada, Hybrid or...  ...per week onsite.Candidates not within...  ...quality, maintainable code that demonstrates...  ...years of professional software engineering experience...  ...including safety evaluations, iterative testing... 
    Full time
    Work at office
    Remote work
    2 days per week

    Wolters Kluwer

    Princeton, NJ
    17 hours ago
  • $65 - $73 per hour

     ...advanced), Git (advanced), AI coding tools (advanced), API Integration...  ...performance of AI-powered software development tools. You will engineer evaluation frameworks and automate benchmarking processes to ensure...  ...Inc 5000 Best Workspaces in US (2025 & 2024) and Glassdoor... 
    Hourly pay
    Contract work
    Remote work

    Akraya, Inc.

    San Diego, CA
    26 days ago
  • $173k - $251k

     ...looking for a Senior Software Engineer, AI Platform with...  ...model orchestration, evaluation, and agentic...  ...capabilities, write great code, learn a huge...  ...ahead of us, but don’t rely on...  ...in the “perfect” candidate and encourage you...  ...relevance tuning, benchmarking, and multi-modal... 
    Full time
    Work experience placement
    Work at office
    Local area
    Remote work
    2 days per week
    3 days per week

    Everlaw

    Oakland, CA
    2 days ago
  • $180k - $260k

     ...is a leading AI-powered performance...  ...Role: As a Software Engineer at Liftoff...  ...Continuously evaluate emerging tools...  ...participate in code reviews and mentorship...  ...first company with US hubs in Redwood...  ...based on the candidate's location and...  .... We benchmark compensation based... 
    Full time
    Remote work
    Worldwide

    Liftoff

    Nevada
    4 days ago
  • $128k - $219k

     ...As a Senior Software Engineer at Coder, you’ll build and evolve...  ...Contribute to design reviews, code reviews, and technical...  ...leading platform for AI development...  ...Ashby and Brainerr help us review inbound applications by surfacing candidates who best match the role... 
    Full time
    Local area
    Remote work

    Coder

    Remote
    6 days ago
  • $190.8k - $267.1k

     ...Compliance Engineering team is responsible...  ...a GRC Software Engineer to...  ...HR, finance, code, and ticketing...  ...to candidates, we share base...  ...ranges for all US-based job postings...  ...location, benchmarked against...  ...intelligence (AI). You will have...  ...to evaluate your application... 
    Full time
    For contractors
    Work experience placement
    Flexible hours

    Reddit

    Remote
    19 days ago
  • $237.6k - $401.7k

     ...5Team: Machine Learning and AI, Software and ServicesLocation: CupertinoWeekly...  ...As part of the Siri Agentic Evaluation Engineering organization, you will help...  ...harnesses that let us evaluate Siri reliably and at...  ...workflows (e.g., data generation, code generation, or agentic... 
    Relocation

    Apple

    Cupertino, CA
    3 days ago
  • $96k - $169.3k

     ...Location Remote, US Scheduled...  ...Brain Research and AI in...  ...developing plans for benchmarking and evaluation of the tools in...  ...C-BRAIN Senior Software Development Engineer builds and supports...  .... The ideal candidate is a production...  ...of AI-generated code in sandboxed containers... 
    Full time
    Contract work
    For contractors
    Work experience placement
    Work at office
    Local area
    Remote work

    Washington University in St. Louis

    Remote
    4 days ago
  • $124k - $168k

     ...best-of-breed software applications already...  ...Software Engineer (Agentic AI), to build the...  ...many lines of code you write. It...  ...environments. Evaluate the agent end-...  ...speaking to candidates with more or...  ...approach to base pay benchmarking and pay band...  ...off Plan (for US exempt... 
    Full time
    Work at office
    Immediate start
    Remote work
    Flexible hours
    1 day per week

    Dotmatics

    Remote
    5 days ago
  •  ...About IRIS IRIS Software Group is one of the...  ...a hands-on senior engineering role where you’ll...  ...technologies, helping us create secure,...  ...towards becoming an AI-first organisation...  ...Infrastructure as Code. Work closely...  ...application profiling, benchmarking and performance... 
    Permanent employment
    Full time
    Work at office
    Remote work

    IRIS

    Remote
    13 days ago
  • $96k - $192k

     ...Services and CF Benchmarks - has spent...  ...next generation US-based global...  ...infrastructure as code with Terraform...  ...professional software engineering experience...  ...judgment with AI tools. You use...  ...team, assessing candidates in a manner consistent...  ...assessments evaluate competencies... 
    Full time
    Local area
    Worldwide
    Shift work

    Kraken

    Remote
    12 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Software Engineer - AI Code Evaluation & Benchmarking (US candidates only) [Remote]. Be the first to apply!