AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$20 - $30 per hour
...ROLE We evaluate how well advanced AI coding agents solve real engineering problems. An agent... ...on its own for 80 to 250 steps. A trajectory is the full record of that run — every... ...at yet and judge it step by step. • Review — Take an existing annotation, written...SuggestedHourly payFull timeContract workImmediate startRemote workBespoke Labs
Gaithersburg, MD3 days ago- ...document completeness. Scan, digitize, and upload personnel records into designated electronic document management systems. Review scanned documents to verify image quality, completeness, legibility, and accuracy. Index and classify electronic records according...SuggestedFull timeContract workFor contractorsWork at office
Eigennet
Rockville, MD4 days ago $10k
...to take the first step to becoming an entry level Border Patrol Agent. USBP is hiring immediately to fill full-time, entry-level... ...employment site, to complete your application. Be certain to review ALL details of the job opportunity announcement and follow all instructions...SuggestedFull timeWork experience placementImmediate startRelocationNight shiftUS Customs and Border Protection
Kensington, MD2 days ago- ...Hmong Chinese Language Document Reviewer Employment Type: Full Time, Mid-Level Department: Legal As a CGS Hmong Chinese Document Reviewer you will play a key role in supporting various aspects of the company's litigation portfolio including eDiscovery activities...SuggestedFull timeWork at officeFlexible hours
$750 per week
ApexFocusGroup collaborates with research firms, universities, and trusted brands to gather genuine consumer insights. We help connect everyday people with flexible remote market research opportunities that influence real products, services, and customer experiences. Role...SuggestedPart timeCasual workRemote workFlexible hoursAFG
Rockville, MD2 days ago$55.12k - $57.41k
...same, however a typical day in the life of an Asset Protection Agent will likely include the following responsibilities: Resolve theft... ...related to this job posting that are specific to your state. Review the Career Site FAQ’s for relevant information and guidelines....Hourly payFull timeLocal areaFlexible hours$55k - $75k
...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State... ...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop...For contractorsFlexible hoursJames Reid Ltd
Gaithersburg, MD1 day ago$40k - $45k
...Responsive RecruiterJames Reid - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual to join our team as a Receptionist - State Farm Agent Team Member. Our ideal candidate...For contractorsWork at officeJames Reid Ltd
Rockville, MD2 days ago$68k - $75k
...Job Description Job Description Position: Quality Control Data Reviewer ISalary Range: $68,000- 75,000Location: Germantown, MDAbout PharmaronPharmaron is a global contract research and manufacturing organization (CRO/CDMO) helping pharma and biotech companies bring...Contract workTemporary workWorldwideFlexible hoursPharmaron
Germantown, MD16 days ago- Telework Eligible Yes Major Duties Advises customers on the latest Document Services available, e.g., printing, document automation, and equipment management technologies. Provides advice and assistance to customers and serves as agency representative. Gathers...Full timeWork at officeRemote work
Defense Logistics Agency
Gaithersburg, MD1 day ago $18 - $21 per hour
Job Description Job Description The Filing Clerk maintains the physical and electronic records across the Accounts Payable, Accounts Receivable, and Commercial Real Estate functions . Working across multiple departments, this individual maintains filing systems, ...Hourly payWork at officeChuck Levin's Washington Music Center
Silver Spring, MD6 days ago$40k - $45k
...Responsive Recruiter James Reid - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our customers. We are currently seeking a dynamic individual to join our team as a Receptionist - State Farm Agent Team Member. Our ideal candidate...For contractorsWork at office- Arlluk Technology Solutions (ATS), a Koniag Government Services company, seeks an experienced Help Desk Technician to support ATS and government customer in Silver Spring, MD. This is an onsite role with a comprehensive benefits package. Key functions include Tier I/II...
Koniag Government Services
Silver Spring, MD3 days ago $50 - $55 per hour
...Residential Security Agent II (Hybrid) Global Guardian protects and delivers employees and families from political, environmental,... ...designated residence. Perform local area travel advances to review departures, arrivals, movements, routes, and emergency actions and...Hourly payFull timeLocal areaWorldwideGlobal Guardian
McLean, VA1 day ago$45 - $65 per hour
...Executive Protection Agent Global Guardian protects and delivers employees and families from political, environmental, and bad actor... ...domestic assignments. Perform local area travel advances to review departures, arrivals, movements, routes, and emergency actions and...Hourly payPart timeWork at officeLocal areaWorldwideFlexible hoursGlobal Guardian
McLean, VA10 hours ago$20 - $25 per hour
...We're Hiring! Responsive recruiter Position Overview State Farm Insurance Agent located in Bethesda, MD is seeking an outgoing, career-oriented professional to join their team. As a State Farm team member for David Munson Agency, you will build and develop customer...Hourly payFor contractorsInternshipWork at officeDavid Munson - State Farm Agent
Bethesda, MD1 day ago- ...Agent Team Member Position Looking for a part-time job that provides meaningful work and competitive compensation? Consider a position in a State Farm Agent's office. Responsibilities Establish customer relationships and follow up with customers, as needed....Hourly payPart timeFor contractorsWork at officeFlexible hours
$40k - $60k
...want to meet you and talk about the opportunity we have for you. Come work with an energetic, fun team at Jamie Schad - State Farm Agent! About Our Agency Our mission is to help people manage the risks of everyday life, recover from the unexpected and realize their...For contractorsWork at officeState Farm
Olney, MD3 days ago- ...serving the community since 1994 and is backed by decades of dedication, experience, and family legacy. As a second-generation State Farm agent — and now with my daughter also following in my footsteps — insurance and service run deep in our family. We’ve built a trusted,...Full timePart timeWork at officeLocal areaFlexible hours
Mickey Parker - State Farm Agent
Silver Spring, MD12 hours ago $36k - $50k
...business, we encourage you to apply. Zach Armiger State Farm is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. Compensation: $36,000.00 - $50,000.00 per year Zach Armiger - State Farm AgentFlexible hours$18 per hour
...SUMMARY A detail-oriented data curator responsible for organizing, entering, reviewing, and managing scientific information. This role requires strong scientific literacy, sound judgment, and the ability to accurately interpret and categorize technical content. This...Hourly payFull timeTemporary workFor contractorsWork at officeImmediate startCHICKASAW NATION INDUSTRIES INC
Rockville, MD1 day ago- ...these questions, working for a State Farm independent contractor agent may be the career for you! State Farm agents market only State... ...with customers, as needed. Use a customer-focused, needs-based review process to educate customers about insurance options. Develop...For contractorsWork at officeRelocation package
$149.8k - $171k
...Overview Manager, Product Management – Agent Enablement & Coaching Product Management... ...Frontline Coaching: Shift from delayed reviews to "just-in-time" workflows, reducing... ...targeted, automated coaching alerts. Pioneer AI for Servicing Monitoring: Run experiments...Full timePart timeLocal areaImmediate startShift workCapital One
McLean, VAa month ago$50k - $60k
...Receptionist - State Farm Agent Team Member Jay Jarvis - State Farm Agency | Germantown, MD $50,000 - $60,000 per year | Full-time | In-office Jay Jarvis - State Farm Agency in Germantown, MD is looking for a friendly, professional and organized Receptionist to be...Full timeFor contractorsWork at officeLocal areaJay Jarvis - State Farm Agency
Germantown, MD10 days ago- Job Description Job Description Benefits: Bonus based on performance Flexible schedule Paid time off Training & development You May Be a Great Fit as an Acquisitions Specialist at Mensah Agency if: You’re the person people call first when life goes...Work at officeFlexible hours
Ray Mensah - State Farm Agent
Silver Spring, MD20 days ago - NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
US Postal Service
Garrett Park, MD1 day ago $26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...Hourly payFull timeCurrently hiringWork at officePostal Jobs Source (a division of Labor Services)
North Bethesda, MDmore than 2 months ago- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
US Postal Service
Potomac, MD1 day ago $18 - $25 per hour
...Telemarketer - State Farm Agent Team Member Successful State Farm Agent is seeking a qualified professional to join their winning team for the role of Telemarketer - State Farm Agent Team Member. We seek an energetic professional interested in helping our business grow...Hourly payFor contractorsWork at officeFlexible hours$18 per hour
Service Deli ClerkLove fast-paced work and leaving lasting impressions with every customer? In our stores, a positive attitude, teamwork and a passion for great service turn every shopping trip into an experience our customers won't forget.Position Summary: The Service...Hourly payLocal areaAfternoon shiftGiant Food
Gaithersburg, MD4 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
