SWE Expert

You'll be the engineering judgment behind how frontier coding models get trained and evaluated. The work centers on reading AI coding transcripts and pull requests and making calls that hold up to scrutiny: why one implementation is better than another, where a test is weak or can be passed without solving the problem, and what a diff actually changes. You'll work with the latest state-of-the-art models inside AI coding agents, and you'll be expected to know where each recent release falls short and where it improves.

There are two tracks. Independent contractors review code and model transcripts and write clear, defensible judgments, often working within rubric, calibration, and consensus workflows. Pod leads do that same review work and also run a small team of engineers. They turn researcher intent into rubrics and tasks the pod can execute consistently, push back when a spec would produce gameable labels, and coach reviewers through real conceptual gaps such as reward hacking. Future pod leads will be promoted from the contractor group, so strong individual reviewers have a clear path forward.

This is a fully remote, exclusive contract engagement requiring 40 hours per week for 5 months. You cannot hold a concurrent job or work on other AI data platforms during this engagement. This is not a part-time or flexible 10–40-hour opportunity. The process is a 20-minute intro call with our team, a 45-minute interview with Claude, a 90-minute CodeSignal assessment, and a 2-hour trial task.

What you'll do

• Review AI-generated code, agent transcripts, and PRs, then write clear, defensible judgments about correctness, quality, and what each change actually does
• Compare competing implementations and explain in writing why one is better, including tradeoffs, edge cases, and hidden regressions
• Identify weak or gameable tests and spot when a model has satisfied the checks without solving the underlying problem
• Create, QA, and calibrate model-testing tasks and rubrics so that labels stay consistent across reviewers
• Use and maintain thoughtful AI coding agent setups to stress-test the latest SOTA models and document how different releases fail or improve
• (Pod leads) Translate researcher goals into rubrics and actionable tasks, and flag specs that would lead to reward hacking or noisy labels
• (Pod leads) Train, calibrate, and performance-manage a remote pod, and tell the difference between inattention and genuine conceptual gaps so you can coach the gaps directly

Who we're looking for

• You have strong software engineering expertise and can read an unfamiliar PR or transcript and judge it quickly and accurately. This is the core bar for both tracks.
• You write clearly and precisely, and your code-review judgments hold up when someone else checks them
• You've done at least one of the following: created or QA'd AI model-testing tasks at a reviewer or verifiable-coding level, with fluency in rubrics, calibration, and consensus; or used AI coding agents heavily and formed real opinions about how models fail. Having both is a strong plus.
• You run a sophisticated, tasteful AI coding agent setup and you follow the latest model releases closely enough to explain how they differ
• You understand how tasks and rewards get gamed. Familiarity with RLHF, reward models, reward hacking, or test, CI, and verification infrastructure is a plus.
• For the pod lead track, you've trained, calibrated, and performance-managed engineers, ideally on remote teams. Prior lead or QA roles at an AI training or evaluation company, or experience designing hiring screens, are a plus.
• You can commit to this engagement for 5 months

Details

• Pay: $150 / hr
• Workload: 40 hrs/week — exclusive
• Duration: 5 months
• Start: Immediately
• Type: W-2 Contract

Massive (Do a Huddle, Inc.) runs a short intro call and interview and names the client before you commit. Never pay to get work.