Software Engineer - AI Code Evaluation & Benchmarking (US candidates only) [Remote]
AuraOne Human Data
- Remote job
Software Engineer - AI Code Evaluation & Benchmarking (US candidates only) is a remote engineering review track for evaluating production code, debugging traces, and developer-facing AI outputs against real-world correctness standards. Reviewers reproduce failures, write the unit test the model should have written, and explain the fix so the modeling team can target the gap.
Why this role matters
Engineering model quality lives or dies on whether the generated code actually compiles, passes tests, and handles edge cases. AuraOne pairs experienced engineers with the modeling team to grade outputs the way a code reviewer would.
Responsibilities
- Run and reproduce candidate code outputs in a sandboxed environment for Software Engineer - AI Code Evaluation & Benchmarking (US candidates only) assignments.
- Grade software engineer ai code evaluation benchmarking engineering review solutions for correctness, style, and edge-case handling.
- Write minimal failing tests that demonstrate the bug a model output missed.
- Compare paired solutions and rank them with a written rationale tied to the rubric.
- Tag failure modes (compile error, runtime crash, off-by-one, security issue) with severity scores.
- Document recurring code-generation failures so the modeling team can target them.
- Calibrate against gold-standard reviews to keep inter-rater agreement above target.
Qualifications
- Strong day-job engineering experience — you can read, run, and debug unfamiliar code for Software Engineer - AI Code Evaluation & Benchmarking (US candidates only) work.
- Comfort writing concise unit tests that capture a single failure mode.
- Familiarity with a testing framework. pytest, Jest, or JUnit. Go test, RSpec, or whatever you use.
- Clear written reasoning — your review note has to convince another senior engineer.
- Reliable async availability for at least 10 hours per week.
- Prior code-review or technical-interview-grading experience is a plus.
Example tasks
- Reproduce a generated engineering solution to a coding task, run the test suite, and grade it.
- Write the smallest failing test that demonstrates a model's edge-case bug.
- Compare two paired solutions and rank them with a written rationale tied to the rubric.
- Triage a security issue surfaced by a model and document the patch the model should have produced.
Nice to have
- Open-source contributions or a public portfolio that demonstrates production-quality code.
- Experience with the target language's standard tooling, linters, and idiomatic style guides.
- Familiarity with security-review checklists (OWASP, CWE) and AppSec patterns.
Skills
- Code review
- Debugging
- Unit testing
- Software engineering judgment
- Software Engineer AI Code Evaluation Benchmarking engineering review
- Python
- Git
- Docker
- Legal & Law Services
Work model
Remote — US-eligible. Remote · Independent specialist contractor. Employment type: CONTRACTOR. Applicants must be authorized to work from US.
Compensation
Hourly rate confirmed after the interview process.
Application process
Apply through AuraOne's specialist intake for role-specific routing and review. Final project scope, schedule, and contractor terms are confirmed before placement.
$75 - $105 per hour
...Opportunity Review Python code, tests, and execution logs from AI coding tasks to... ...organization to evaluate how coding agents perform on software engineering tasks. A failing... ...with coding benchmarks such as SWE-bench... ...Process Shortlisted candidates will need to complete...SuggestedHourly payImmediate startRemote work$197.3k - $225.1k
...Job Description AI Engineer 4 (AI Foundations: Benchmarking, Evaluation, and... ...learning — position us to be at the forefront... ...deploy, and support AI software components including... ...translation The Ideal Candidate: You love to... ...California Police Code Article 49,...SuggestedFull timePart timeLocal area$90 per hour
...Pay: Up to $90+/hr Experienced software engineers review and score AI-written code for frontier AI research teams. Remote and flexible, up to 40 hours... ...professional software engineering experience; you live in the US or Canada and can work as an independent contractor...SuggestedHourly payTemporary workFor contractorsLive inRemote workVisa sponsorshipFlexible hours- ...About Us Based in San Francisco... ...for frontier AI labs and a trusted... ...specialize in software engineering, logical... ...Software Engineering evaluator, you will... ...for training, benchmarking, and advancing... ...includes curating code examples,... ...Location: Candidates must be based...SuggestedContract workFor contractorsFlexible hours
- ...About Us Based in San Francisco... ...for frontier AI labs and a... ...specialize in software engineering, logical reasoning... ...Engineering evaluator, you will create... ...for training, benchmarking, and advancing... ...curating code examples, providing... ...) Location: candidates must be based...SuggestedContract workFor contractorsFlexible hours
$139.2k - $235.2k
...more secure software faster. The... ...: we embrace AI as a core productivity... ...future with us as we build... ...Backend Engineer on GitLab's Security... ...Factory: Code Security team... ...develops, evaluates, and ships those... ...against benchmark applications... ...interest from candidates with varying...Full timeLive inRemote workFlexible hours$80 - $100 per hour
...Senior Python Developer (AI Evaluation & Benchmarking) An enterprise client... ...evaluating AI-generated code, and helping improve the... ...opportunity for experienced software engineers who enjoy solving complex... ...the client. Qualified candidates will receive an Upwork contract...Hourly payWeekly payContract workRemote work10 hours per week- ...About Us Our mission is to raise... ...was founded by engineers and researchers... ...next generation of AI. Our platform powers... ...rigorous evaluations that go beyond benchmarks. We've run a profitable... ...Engineer, Coding Evaluation & Training... ...intersection of software engineering and...Full time
- ...clients across the US. What makes us different... ...out. Title: Software Engineer - AI Evaluation & Automation... ...harnesses, automate benchmark runs, and help make... ...experience using AI coding tools and agentic harnesses... ...zone. Not accepting candidates with more than a two...For contractorsRemote workFlexible hours
$10k
...LabsRamp Labs is Ramp's AI research group.... ...runs through a benchmark. Which model to... ...review, invoice coding, month-end close,... ...decisionsStrong Python and software engineering; your eval code... ...benchmark or evaluation research (NeurIPS... ...and level at which the candidate is hired.Full timeWork at officeHome officeFlexible hours$144.7k - $221.4k
...unprecedented scale. Join us to help deliver... ...The Evaluation team builds and... ...clear feedback for engineering and leadership,... ...driving software performance at... ...system design, code reviews, testing... ...Experience leveraging AI-assisted... ...the successful candidate is expected to...Full timeWork at officeLocal areaWork from homeRelocationRelocation packageFlexible hours$140k - $175k
Senior Software Engineer (Hybrid - US) Boston, Massachusetts, United States... ...Software Engineer evaluates technical requirements... ...level engineers through code reviews, technical... ...opportunities. We base our candidate selection on one's... ...intelligence (AI) tools (including Generative...Work at officeRemote work- ...your career with us, experience our caring... ...& Insurance AI Evaluation Engineer PRIMARY PURPOSE:... ...to U.S. based candidates. ESSENTIAL FUNCTIONS... ...evaluation (benchmark and golden... ...Experience 4+ years in software, ML, or data... ...control, CI/CD, code review). Practical...Full timeLocal areaRemote work
- ...Epoch AI is looking for a Software Engineer, Cybersecurity Benchmarks who will help us measure how capable frontier AI models... ..., securing our evaluation environments against... ...releases quickly. An ideal candidate for this role might... ...familiarity with AI coding harnesses. Your work...Full timeRemote workFlexible hours
$250k - $350k
...is the Work AI platform that... ...’t writing code anymore. It’... ...Manager of AI Evaluations, you will... ...scientists and engineers and build... ...experience in software engineering,... ...datasets, benchmark creation, deterministic... ...for Job Candidates and... ...as required. US applicants and...Work at officeHome officeFlexible hours- Software Engineer III Full Time - Remote Cincinnati... ...Innovate with Benchmark Gensuite as a Senior... ...inclusion. Join us and help make... ...clients. The ideal candidate will perform coding, debugging,... ...build processes AI & Emerging Technology... ...feedback Evaluate and integrate AI...Full timeImmediate startRemote workWorldwideFlexible hours
$107k - $160k
...Netherlands. Visit us at We are... ...care about our candidates, employees and... ...passionate Senior Cloud Engineer to join our... ...-as-code best practices... ...: Partner with software engineers and other... ...Experience using AI-powered coding... ...to critically evaluate and validate generated...Full timeRemote workWork from homeWorldwideHome office$98.61k - $167.64k
...experienced and driven Senior-Level Software Engineer to lead and oversee mission-... ...software solutions. Conduct code reviews, unit and... ...****@*****.*** for assistance.Candidate AI Usage PolicyAt ICF, we are... ...are instructed to contact us in advance at...Full timeContract workWork experience placementInterim roleWork at officeRemote work$87.8k - $209.5k
...Clinical Applications Engineering Transform... ...skilled Principal Software Engineer to join our... ...Engineering Productivity & AI Enablement... ...tools, including coding assistants and prompt... ...Role The ideal candidate is first and foremost... ...locations only US: Hiring Range in...Full timeTemporary workWorldwideFlexible hours$99.6k - $174k
Senior Full Stack Engineer, AI Platform & AgentsBuild... ...scale.-Location:: US/Canada, Hybrid or... ...per week onsite.Candidates not within... ...quality, maintainable code that demonstrates... ...years of professional software engineering experience... ...including safety evaluations, iterative testing...Full timeWork at officeRemote work2 days per week$65 - $73 per hour
...advanced), Git (advanced), AI coding tools (advanced), API Integration... ...performance of AI-powered software development tools. You will engineer evaluation frameworks and automate benchmarking processes to ensure... ...Inc 5000 Best Workspaces in US (2025 & 2024) and Glassdoor...Hourly payContract workRemote work$173k - $251k
...looking for a Senior Software Engineer, AI Platform with... ...model orchestration, evaluation, and agentic... ...capabilities, write great code, learn a huge... ...ahead of us, but don’t rely on... ...in the “perfect” candidate and encourage you... ...relevance tuning, benchmarking, and multi-modal...Full timeWork experience placementWork at officeLocal areaRemote work2 days per week3 days per week$180k - $260k
...is a leading AI-powered performance... ...Role: As a Software Engineer at Liftoff... ...Continuously evaluate emerging tools... ...participate in code reviews and mentorship... ...first company with US hubs in Redwood... ...based on the candidate's location and... .... We benchmark compensation based...Full timeRemote workWorldwide$128k - $219k
...As a Senior Software Engineer at Coder, you’ll build and evolve... ...Contribute to design reviews, code reviews, and technical... ...leading platform for AI development... ...Ashby and Brainerr help us review inbound applications by surfacing candidates who best match the role...Full timeLocal areaRemote work$190.8k - $267.1k
...Compliance Engineering team is responsible... ...a GRC Software Engineer to... ...HR, finance, code, and ticketing... ...to candidates, we share base... ...ranges for all US-based job postings... ...location, benchmarked against... ...intelligence (AI). You will have... ...to evaluate your application...Full timeFor contractorsWork experience placementFlexible hours$237.6k - $401.7k
...5Team: Machine Learning and AI, Software and ServicesLocation: CupertinoWeekly... ...As part of the Siri Agentic Evaluation Engineering organization, you will help... ...harnesses that let us evaluate Siri reliably and at... ...workflows (e.g., data generation, code generation, or agentic...Relocation$96k - $169.3k
...Location Remote, US Scheduled... ...Brain Research and AI in... ...developing plans for benchmarking and evaluation of the tools in... ...C-BRAIN Senior Software Development Engineer builds and supports... .... The ideal candidate is a production... ...of AI-generated code in sandboxed containers...Full timeContract workFor contractorsWork experience placementWork at officeLocal areaRemote work$124k - $168k
...best-of-breed software applications already... ...Software Engineer (Agentic AI), to build the... ...many lines of code you write. It... ...environments. Evaluate the agent end-... ...speaking to candidates with more or... ...approach to base pay benchmarking and pay band... ...off Plan (for US exempt...Full timeWork at officeImmediate startRemote workFlexible hours1 day per week- ...About IRIS IRIS Software Group is one of the... ...a hands-on senior engineering role where you’ll... ...technologies, helping us create secure,... ...towards becoming an AI-first organisation... ...Infrastructure as Code. Work closely... ...application profiling, benchmarking and performance...Permanent employmentFull timeWork at officeRemote work
$96k - $192k
...Services and CF Benchmarks - has spent... ...next generation US-based global... ...infrastructure as code with Terraform... ...professional software engineering experience... ...judgment with AI tools. You use... ...team, assessing candidates in a manner consistent... ...assessments evaluate competencies...Full timeLocal areaWorldwideShift work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Software Engineer - AI Code Evaluation & Benchmarking (US candidates only) [Remote]. Be the first to apply!
- junior software developer Remote
- senior software design engineer Remote
- software web developer Remote
- senior software engineer Remote
- labview software developer Remote
- software sales engineer Remote
- software developer fintech Remote
- software support engineer Remote
- security software engineer Remote
- software developer night shift Remote






