AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$114.1k - $268.18k
...new areas of inspiration and expand your capabilities, then consider a career in Advisory.KPMG is currently seeking a Manager, SAP AI & Agent Governance to join our Advisory Technology Organization.Responsibilities:Lead the design and delivery of AI-enabled and agentic...SuggestedH1bLocal area- EY is seeking an experienced Lead AI Software Architect to lead a team of domain experts in designing and implementing AI Agent Control. You will shape architectures for agent identity, authentication, and post-training environments, while driving experimentation and collaboration...Suggested
- Safelite is introducing an AI FNOL scheduling agent, with the Senior Product Analyst responsible for translating client requirements into clear, buildable specs. You’ll work with product, engineering, design, and client-facing teams to drive the agent from definition to...SuggestedRemote job
$34.57 - $46.11 per hour
...every stage of your career. Try new things, learn new skills and discover what you excel at—all from Day One. Job Description Reviews appraisals and reports including desktop valuations from a wide variety of bank channels. Reviewer Appraiser positions review all types...SuggestedTemporary workWork at officeLocal area3 days per weekU.S. Bank
Columbus, OH19 hours ago- Huntington Bank in the Columbus area is seeking a Credit Review Officer for its Consumer line. The role focuses on conducting credit risk reviews and continuous monitoring to ensure compliance with policies and regulatory requirements. The ideal candidate has extensive...SuggestedRemote job
$17.5 - $22.86 per hour
Job Title Seasonal Honors Reviewer Department Strategic Enrollment | Undergraduate Admissions Role Description The Honors Application Reviewer in Undergraduate Admissions within Strategic Enrollment Management (SEM) provides a holistic evaluation source for domestic and...Hourly paySeasonal workLive inWork at officeRemote workFlexible hoursShift workDay shift- ...Procore Submittals as outlined in contract documents Maintain distribution list of responsible parties for various stages of document review process Assign workflow to responsible parties to review vendor documents within Procore or other document control systems...Contract workTemporary workWork at officeLocal area
Quanta Infrastructure Solutions Group, LLC
Columbus, OH19 hours ago $66.41k - $99.61k
...and select option 5 for TTY service. Please indicate to the HNTB agent that you are a candidate seeking assistance in the job... ...employee or applicant for employment. These plans are available for review upon request during normal business hours. Any questions or requests...Full timePart timeFor contractorsWork at officeLocal areaImmediate startHNTB
Columbus, OH19 hours ago$75 - $90 per hour
...homes from a customer's perspective to identify defects in workmanship, quality, completeness, consistency, and professionalism. (Reviews take on average 1 to 1.5 hours to complete) Each review is meant to evaluate the quality of the finished construction. Driving to...Part timeFor contractorsMonday to Friday- The Ohio State University is seeking a Seasonal Honors Reviewer for Undergraduate Admissions in Strategic Enrollment Management. You will conduct holistic reviews for Honors Program applications and enter recommendations into the review system on a weekly basis. This seasonal...Remote jobSeasonal work
$21.32 per hour
...years Experience level: Experienced Randstad is a world leader in matching great people with great companies. Our experienced agents will listen carefully to your employment needs and then work diligently to match your skills and qualifications to the right job...Hourly payPermanent employmentContract workTemporary workWork experience placementShift workColumbus, OH1 day ago$40.25k - $57.66k
...flexibility, and long-term opportunity. What You’ll Do Prepare, review, and quality-check right of way documents and parcel files for... ...Coordinate document execution, notarization, and recording Support Agents and the Project Manager with document requests, reporting, and...Daily paidPart timeWork experience placementRemote workRelocationLong distance3 days per week- RR Donnelley is hiring a Senior Document Production Associate in Columbus, OH. In this hybrid role, you will be responsible for providing word processing and document production services. The position requires a high school diploma and at least 3 years' experience in a ...Hourly payAfternoon shift
- Williams Lea by RRD is seeking a Senior Document Production Associate to provide word processing and document production services for clients. You will assess requests, select proper production processes, and complete tasks to client satisfaction. Role requires advanced...
- A leading document production company is seeking a Document Production Associate for their Columbus, OH office. The role involves managing document production tasks, ensuring quality and meeting deadlines. Candidates should have a high school diploma, proficient MS Office...
- RR Donnelley & Sons is seeking a Document Production Workflow Coordinator to deliver timely and accurate document production in a high-volume environment. The role requires advanced software aptitude for proofreading, edits, formatting, conversions, and quality checks across...Shift work
- Williams Lea by RRD is seeking a Senior Document Production Associate in Columbus, OH to provide word processing and document production services for clients. You will perform production tasks, assess requests, edit and transcribe to client satisfaction, and meet deadlines...
- Neier Inc. is seeking a Clinical Documentation Support Specialist in Columbus, OH. This full-time position plays a key role in ensuring clinical information flows accurately between AndHealth and partners’ EHRs. You will be proactive in maintaining communication, resolving...Full time
- RRD Company in Columbus, OH, is seeking a Senior Document Production Associate to provide word processing and document production services. The role is hybrid, Monday-Friday with flexible start times (10am-7pm or 11am-8pm). The position requires 3+ years in document production...Monday to FridayFlexible hours
$17.74 - $24.47 per hour
RRD is seeking a Senior Document Production Associate for their Columbus, OH, office. This hybrid position involves performing document production tasks and ensuring quality control while interacting with clients and maintaining confidentiality. The ideal candidate should...Hourly payWork at office$18.5 per hour
A global business support services firm is seeking a detail-oriented document review specialist to work hybrid from Columbus, OH. The role includes creating and editing legal documents, handling confidential information, and utilizing advanced MS Office skills. Preferred...Hourly payNight shift$18.75 - $19.75 per hour
...processing company in Columbus is seeking a detail-oriented individual for a document production role. Responsibilities include document review, preparation of spreadsheets, and managing multiple projects to meet deadlines. Ideal candidates must possess a high school diploma...Hourly pay- ...of Courts seeks a Title Management Specialist to assist customers with the issuance of legal documents and vehicle titles. You will review documents, process transactions, and ensure accuracy in title records and notary requirements. You will handle in-person and phone...
- RRD is seeking a detail-oriented document reviewer to support client projects with strong MS Office skills in a hybrid Columbus, OH setting. Responsibilities include creating and editing legal documents, transcribing, scanning, and converting file formats to meet client...
- Williams Lea by RRD is seeking a Senior Document Production Associate to deliver word processing and document production services for clients. You will assess requests, edit and convert documents, and ensure timely, accurate delivery while maintaining confidentiality. The...
$40.25k - $57.66k
ORC in Columbus, OH is seeking a part-time Public Sector Document Specialist to support right of way documentation and project files. The role reports to Theresa Conley and requires proximity to our Columbus office. The position emphasizes accurate document prep, record...Part timeRelocation- HNTB in Ohio is seeking a Document Controls Specialist to administer the document lifecycle for engineering and construction records. You will define standards, manage publication, storage, and retrieval, and ensure controlled documentation for site teams. The role requires...
- RR Donnelley is seeking a Document Production Associate in Columbus, Ohio. This role involves providing word processing and document production services to clients. Candidates should have a high school diploma and at least one year of relevant experience. The ideal candidate...
$59.36k - $89.04k
What We’re Looking For At HNTB you can build a career that matters while creating communities that affect all of us. For more than a century we deliver solutions for some of the largest, most complex infrastructure projects across the country. With our historic growth it...Full timePart time$59.36k - $89.04k
A national infrastructure firm is seeking a Document Control Specialist to manage project documentation and record retention processes. The ideal candidate will have an Associate’s degree or equivalent experience with 2 years in document control functions. This role requires...Full time
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
