Software Engineer - AI Evaluation

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .

Position: Software Engineering, Data Science, and Systems Design Experts

Type: Contract

Compensation: $60–$100/hour

Location: Remote

Role Responsibilities

• Evaluate LLM-generated responses to coding and software engineering queries for accuracy, reasoning, clarity, and completeness.
• Conduct fact-checking using trusted public sources and authoritative references.
• Conduct accuracy testing by executing code and validating outputs using appropriate tools .
• Annotate model responses by identifying strengths, areas of improvement, and factual or conceptual inaccuracies.
• Assess code quality, readability, algorithmic soundness, and explanation quality.
• Ensure model responses align with expected conversational behavior and system guidelines.

Qualifications

Must-Have

• BS, MS, or PhD in Computer Science or a closely related field .
• Significant (3+ years) real-world experience in software engineering or related technical roles.
• Expert in at least two relevant programming languages (e.g., Python, Java, C++, C, JavaScript, Go, Rust, Ruby, SQL, Powershell, Bash, Swift, Kotlin, R, TypeScript, HTML/CSS ).
• Able to solve HackerRank or LeetCode Medium and Hard–level problems independently .
• Experience contributing to well-known open-source projects, including merged pull requests.
• Significant experience using LLMs while coding and understanding their strengths and failure modes.
• Strong attention to detail and comfortable evaluating complex technical reasoning , identifying subtle bugs or logical flaws.

Preferred

• Prior experience with RLHF , model evaluation, or data annotation work.
• Track record in competitive programming.
• Experience reviewing code in production environments.
• Familiarity with multiple programming paradigms or ecosystems.
• Experience explaining complex technical concepts to non-expert audiences.

Application Process (Takes 20–30 mins to complete)

• Upload resume
• AI interview based on your resume
• Submit form

Resources & Support

• For details about the interview process and platform information, please check:
• For any help or support, reach out to: View email address on jobs.jobcopilot.com

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.