Software Engineer - AI Evaluation
$60 - $100 per hourFull-time
Mercor
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Software Engineering, Data Science, and Systems Design Experts
Type: Contract
Compensation: $60–$100/hour
Location: Remote
Role Responsibilities
- Evaluate LLM-generated responses to coding and software engineering queries for accuracy, reasoning, clarity, and completeness.
- Conduct fact-checking using trusted public sources and authoritative references.
- Conduct accuracy testing by executing code and validating outputs using appropriate tools .
- Annotate model responses by identifying strengths, areas of improvement, and factual or conceptual inaccuracies.
- Assess code quality, readability, algorithmic soundness, and explanation quality.
- Ensure model responses align with expected conversational behavior and system guidelines.
Qualifications
Must-Have
- BS, MS, or PhD in Computer Science or a closely related field .
- Significant (3+ years) real-world experience in software engineering or related technical roles.
- Expert in at least two relevant programming languages (e.g., Python, Java, C++, C, JavaScript, Go, Rust, Ruby, SQL, Powershell, Bash, Swift, Kotlin, R, TypeScript, HTML/CSS ).
- Able to solve HackerRank or LeetCode Medium and Hard–level problems independently .
- Experience contributing to well-known open-source projects, including merged pull requests.
- Significant experience using LLMs while coding and understanding their strengths and failure modes.
- Strong attention to detail and comfortable evaluating complex technical reasoning , identifying subtle bugs or logical flaws.
Preferred
- Prior experience with RLHF , model evaluation, or data annotation work.
- Track record in competitive programming.
- Experience reviewing code in production environments.
- Familiarity with multiple programming paradigms or ecosystems.
- Experience explaining complex technical concepts to non-expert audiences.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on jobs.jobcopilot.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted more than 2 months ago
Similar jobs that could be interesting for youBased on the Software Engineer - AI Evaluation in San Francisco, CA vacancy
- ...Software Engineer, Agent Evaluation and Quality Engineering · Full-time · San Francisco; New York Our mission is to automate coding. The first... ...You'll Work On Designing and building best-in-class AI evaluation system: curated datasets, offline replay, scorers...SuggestedFull timeWork at office
Anysphere
San Francisco, CA1 day ago $150k - $250k
About Distyl AI Distyl is an applied AI technology company partnering... ..., we build AI systems using Evaluation-Driven Development—an... ...in production.AI Evaluation Engineers focus on designing and implementing... ...We Require2+ years of software engineering experience Strong...SuggestedWork at office3 days per week$204k - $259k
...dynamics, and state-of-the-art Generative AI to create a training ground for the Waymo Driver. The Simulator Evaluation team faces the ultimate data challenge: How... ...world is "real"? We are looking for a Senior Software Engineer to build the metrics and systems that grade...SuggestedFull timeRemote work$175k - $215k
...dynamics, and state-of-the-art Generative AI to create a training ground for the Waymo Driver. The Simulator Evaluation team faces the ultimate data challenge: How... ...virtual world is "real"? We are looking for a Software Engineer to build the metrics and pipelines that...SuggestedFull timeRemote work$170k - $216k
...powers the Waymo Driver. Our software allows the Waymo Driver to perceive... ...sensors, enabling software engineers like you to develop multi-... ...-critical automation and evaluation frameworks that establish the... ...of experience in industrial AI applications involving the creation...SuggestedFull timeRemote work- ...mission is to organize human intelligence to power the AI economy. We partner with leading AI labs and enterprises... ...or London offices. About the Role As a Senior Software Engineer (AI Data & Evaluation) at Mercor, you will be at the core of building the data...Full timeWork at officeRelocation package
$208k - $312k
...team behind Next.js, v0, and AI SDK, we create products that... ...exceptional developer experience.Now, software is entering a new era, and... ...for a Senior (IC4) software engineer with a strong security... ...deployed full-time with v0, and is evaluated as much on shipped product...Full timeWork from homeWorldwideFlexible hours$127k - $223k
...Description Waabi, founded by AI visionary Raquel Urtasun, is... .... To learn more visit: The Evaluation Algorithms team is responsible... ...realistic closed-loop simulation engine built with the latest in... ...Python programming and strong software engineering fundamentals with...Full timeWork at officeWork from homeFlexible hours$117.2k - $176.7k
...SalesforceSalesforce is the #1 AI CRM, where humans with agents... ...ExperienceOur Public Cloud engineering teams are responsible for innovating... ...of 3 years of backend software development experience (Python... ...our recruiters assess and evaluate candidates’ resumes and qualifications...Full timeWork experience placement$148.5k - $223.9k
...SalesforceSalesforce is the #1 AI CRM, where humans with agents... ...ExperienceOur Public Cloud engineering teams are responsible for innovating... ...of 5+ years of backend software development experience (... ...help our recruiters assess and evaluate candidates’ resumes and qualifications...Full timeWork experience placement$120k - $170k
...CO / San Francisco, CAMission & Program Engineering - Software Solutions /Full time /On-siteWanna join... ...this role is intentionally wide as we evaluate individuals based on their unique experience... ..., IoT connectivity, on-orbit AI, national security missions, and more....Full timeTemporary work$122k - $240.5k
Position Summary Software Engineer III with Agentic AI Experience, AI & Engineering/Engineering as a ServiceAgentic AI is moving from experimentation... ...as orchestration, tool integration, state management, evaluation, guardrails, and human-in-the-loop controlsAt least 1...Work at officeLocal areaVisa sponsorshipFlexible hoursShift work$170k - $240k
...architecture. Hence, we are growing the engineering team and looking for... ...and implement new software features to support our fast... ...package.About us: Sigma is the AI Apps and agentic analytics platform... ...personal data for the purposes of evaluating your candidacy for employment...Full timeWork at office- ...reliable on-demand, logistics engine for delivery. We are continuing... ...-functionally, as well as evaluating and executing on trade-offs to... ...tech stacksProficiency in using AI coding tools (e.g., Claude Code, Codex, Cursor) in the full software development lifecycle, including...Hourly payWork at officeLocal areaRemote workFlexible hours
$230k - $390k
...human customer experiences with AI. We are primarily an in-... ...development is an evolution of software development that requires new... ...ll design systems that enable engineers and customers to build... ...job description. We strive to evaluate all applicants consistently without...Full timeFlexible hours$148.5k - $223.9k
...DetailsAbout SalesforceSalesforce is the #1 AI CRM, where humans with agents drive... ...looking for an experienced and hands-on software engineer to join our team to build and scale the... ...tools to help our recruiters assess and evaluate candidates’ resumes and qualifications throughout...Full time$208k - $312k
...team behind Next.js, v0, and AI SDK, we create products that help... ...developer experience.Now, software is entering a new era, and the... ...We are looking for a Software Engineer to join our Platform team. You... ...and record-keeping requirements.Evaluate, integrate, and help operate observability...Work at officeRemote workWork from homeWorldwideMonday to FridayFlexible hours$153k - $376k
...into code, or iterating with AI. From idea to product, Figma... ...infrastructure and platform-minded engineers to contribute to our agent... ...:Evals for design: Building evaluation frameworks for design... ...you if you have:5+ years of Software Engineering experience, especially...Minimum wageFull timeLocal areaRemote workFlexible hours$216k - $270k
Software is eating the world, but AI is eating software. We live in unprecedented times - AI has the potential... ..., fine-tuning, and production evaluations. This is the main innovation that’s... ...products include the Generative AI Data Engine, SGP, Donovan, and others that...Full timeLive in$172.5k - $260.1k
...SalesforceSalesforce is the #1 AI CRM, where humans with agents... ...platform that enables engineering teams to deliver reliable and... ...observable, and metrics-driven software releases.Reliability measurement... ...help our recruiters assess and evaluate candidates’ resumes and...Permanent employmentFull time- ...patients worldwide.We’re a team of engineers, clinicians, and innovators... ...of PositionAs a Senior Software Engineer - Simulation & ML Platform... ...for robotic motion AI. You will work closely with Simulation... ...development, sim-to-real evaluation, and machine-learning experimentation...Local areaWorldwideFlexible hours
$147k - $210k
...experience developing and maintaining software using C++.Experience... ...distributed services.Experience using AI tools for software... ...debugging).Google's software engineers develop the next-generation technologies... ...and building a new policy evaluation platform and extending...$130k - $220k
...Analysis is the leading independent AI benchmarking and insights company. They help engineers, enterprises, investors, media,... ...role is best described as an AI Evaluation Engineer / Technical Generalist. It is not a traditional software engineering role and not a pure ML...Full timeWorldwideAurora Jobs ApS
San Francisco, CAmore than 2 months ago$187k
...enable data scientists and ML engineers to develop, train, deploy,... ...and efficiently.As a Senior Software Engineer on the Machine Learning... ...machine learning and emerging AI workloads, including model... ...inference, foundation-model access, evaluation, and agentic orchestration....Full timeWork at officeLocal areaRemote work$160k - $200k
...about ourTotal Rewards philosophy.AI is a fundamental part of how... ...Platform and Infrastructure Engineer to join our ML Platform team and... ...passionate about developing software, developing and documenting optimal... ...and sharing best practices to evaluate and scale AI use safely across...Full timeWork at officeLocal area2 days per week3 days per week- ...reliable on-demand, logistics engine for delivery! We're looking for... .... That portfolio includes our AI first reporting and authoring... ...self-serve tooling for creating, evaluating, and improving AI... ...years of industry experience in software engineeringStrong backend fundamentals...Hourly payWork at officeLocal areaRemote workFlexible hours
$250k - $300k
...placing the best product managers, software, and hardware talent at... ...Title of Role: Senior Software Engineer, Backend (Infrastructure)... ...client is a rapidly growing AI infrastructure company building... ...large-scale data generation, evaluation pipelines, and human-in-the-loop...Work at officeRemote workRecruiting from Scratch
San Francisco, CA5 days ago- ...for enterprises to build and orchestrate AI workforces. Our AI workers don't just... ...versatile and highly skilled Senior Backend Software Engineer to join our team. This person will own... ...your personal data for the purpose of evaluating and selecting you as a candidate for...WorldwideShift work
Happy Robot
San Francisco, CA3 days ago - ...Frontend Software Engineer Title of Role: Frontend Software Engineer Location: San Francisco... ...the intersection of labor markets and AI research. This organization... ...Contribute to systems that enable training, evaluation, and scaling for leading AI labs. Ideal...Work at office
Recruiting from Scratch
San Francisco, CA5 days ago $150k - $250k
...Senior Full Stack Software Engineer Location: San Francisco, CA Company Stage of Funding: Series A AI Company ($17M Raised) Office Type: Onsite (5 Days Per Week) Salary:... ...-powered interviews, helping organizations evaluate candidates faster while providing deeper hiring...Work at officeImmediate startRecruiting from Scratch
San Francisco, CA3 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Software Engineer - AI Evaluation. Be the first to apply!
Related searches
- software developer fintech San Francisco, CA
- startup software engineer San Francisco, CA
- financial software developer San Francisco, CA
- junior software developer remote San Francisco, CA
- software engineer San Francisco, CA
- software data engineer San Francisco, CA
- freelance software developer San Francisco, CA
- software developer internship no experience San Francisco, CA
- part time software developer San Francisco, CA
- associate software developer San Francisco, CA