Senior Software Engineer, AI Evaluation and Coding Agents

Role Overview

Help improve AI coding agents by reviewing their work across real-world software repositories. You will assess whether an agent understands a task, selects an appropriate approach, and produces code that is correct, robust, and maintainable. Your technical judgment and precise feedback will help create the evaluation signals used to strengthen model performance.

Key Responsibilities

• Design, refine, and iterate on rubrics and evaluation criteria for preference data.
• Review and label data to a high quality standard, identifying subtle issues that may be missed by others.
• Build and maintain pipelines and infrastructure for data generation, collection, and evaluation workflows.
• Turn findings from data work into clear written updates, recommendations, and documentation for the team.
• Work closely with researchers and engineers to convert qualitative technical judgment into scalable processes.

Qualifications

• At least 5 years of hands-on software engineering experience.
• Strong proficiency in Python, TypeScript or JavaScript, Go, or another major production language.
• Experience working in substantial, real-world codebases.
• Strong code-review skills and technical judgment.
• Ability to clearly explain why an implementation is correct, incorrect, or could be improved.
• Strong written communication skills.
• Experience using modern LLMs or AI coding tools.
• Experience with LLM evaluation, coding agents, RLHF, preference data, rubric design, or post-training is helpful but not required.

Work Terms

• Remote independent-contractor engagement, available only to candidates located in North America.
• Commitment of 20 to 40 hours per week, including at least 6 hours of overlap with Pacific Time each day.
• Approximately 3 months in duration, with a start date as soon as possible.

Compensation

• $100 to $150 per hour.
• Please provide a specific hourly-rate expectation as part of your application.

Application Process

• Complete an approximately 25-minute AI interview.
• Complete an approximately 30-minute practical code and AI evaluation exercise focused on reviewing AI-generated code, rather than competitive programming or algorithm puzzles.
• Meet with the hiring manager for an approximately 20-minute interview.