AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
- ...GenScript Biotech Corp. in Piscataway, NJ is seeking a Senior AI Engineer to design, develop, and deploy enterprise AI Agents and integrations across ERP, CRM, OA, supply chain, and order management systems. You will build capabilities such as task planning and workflow...Suggested
$25 per hour
About the Role Join our distributed team and help evaluate online platforms through interactive testing assignments. This is a flexible, remote opportunity with full training provided and no prior experience required. Reel Edge is a gambling technology company ...SuggestedRemote workWork from homeFlexible hoursJoveo
Highland Park, NJ1 day ago$150k - $180k
...approximately 6,165 employees and has served more than 200,000 customers across over 100 countries and regions. Job Title: Enterprise AI Agent Development Engineer Supervisor: Head of AI for Industry, Digital & Intelligent Technology Center Location: Piscataway, NJ...SuggestedTemporary workWork experience placementFlexible hoursGenScript
Piscataway, NJ2 days ago- ...GenScript USA Inc. in New Jersey invites an Enterprise AI Agent Development Engineer to join our AI & Industry team onsite in Piscataway. You will design, develop, test, and optimize enterprise AI agents that automate cross‑system business processes. You will leverage...Suggested
- ...East Brunswick, NJ is urgently seeking 2 experienced Senior Tax Reviewers to join our growing team. This is an exciting opportunity to... ...federal and multi-state tax compliance ~ CPA license or Enrolled Agent (EA) designation strongly preferred ~ Proficiency in tax...Suggested
Sai Cpa Services
East Brunswick, NJa month ago $22 - $32 per hour
This QA Auditing Reviewer role is responsible for thoroughly auditing production records, determining batch disposition (Release, Hold, Reject), and executing those decisions in systems like JDE while ensuring held or rejected batches are properly controlled and stored...Hourly payTemporary workLocal areaFlexible hours$18 - $25 per hour
...family. World-class facilities and the technology you need to thrive - in our offices or yours. Job Summary The Warehouse Agent is responsible for loading and unloading deliveries, documenting inventory, organizing stock, and verifying orders for stocking and...Hourly payWorldwideFlexible hoursWeekend workSHI International
Piscataway, NJ3 days ago$85k - $110k
...Description Job Description Established steel erection company in the New York metro area, around for 30 years, is adding a Document Reviewer at its Somerset, NJ office. Ideal candidate: Has document review or document control experience: submittals, contracts, shop...Work at officeRoyalty Talent Group
Somerset, NJ7 days ago$18 - $25 per hour
...Warehouse AgentThe Warehouse Agent is responsible for loading and unloading deliveries, documenting inventory, organizing stock, and verifying orders for stocking and shipping. This role involves processing orders in the shipping system, creating packing slips and labels...Hourly payFlexible hoursWeekend workSHI GmbH
Piscataway, NJ4 days ago- ...document control and management Collect and register all technical document, such as drawings and blueprints in the Companys system Review and update documents for maintenance and quality control Keep other personnel updated on new document versions and how to obtain...
$45k - $55k
Assistant Dispatcher Salary Range: $45,000 to $55,000 Experience in Microsoft Word, Excel and Google Workspace School transportation administration experience preferred Routing software experience preferred Experience in non-public school student tracking...- We are offering a $200 sign-on bonus for this area!! Flexible part time jobs now available in your area! Perfect for stay at home Moms and Dads! Looking for an independent part time job while your children are in school? As your children go back to school this fall, ...Extra incomePart timeRelocation packageFlexible hoursShift work
RetailData
East Brunswick, NJ1 day ago - ...Position Overview State Farm Insurance Agent located in Somerset, NJ is seeking an outgoing, career-oriented professional to join their team. As a State Farm team member for Vipul Verma - State Farm Agent, you will build and develop customer relationships within the...Hourly payFor contractorsWork at officeLocal area
State Farm
Somerset, NJ2 days ago $17 - $21 per hour
A leading consulting firm is seeking an entry-level Data Information / Input Assistant to support their team from home. The role involves updating and verifying data, organizing digital files, and ensuring data privacy. The ideal candidate is detail-oriented and willing...Remote jobHourly pay- About the Role Join our distributed team and help evaluate online platforms through interactive testing assignments. This is a flexible, remote opportunity with full training provided and no prior experience required. Reel Edge is a gambling technology company focused ...Remote workWork from homeFlexible hours
Reeledge
Jamesburg, NJ20 hours ago $1,000 per month
...instructions. You can work when it’s convenient. Support is very helpful and responsive.” – Anonymous, Glassdoor (2025) Read more reviews on Glassdoor , Trustpilot , and Google . Equal Opportunity Statement Little Wheel LLC is proud to be an Equal...Extra incomeTemporary workSecond jobCurrently hiringWork from homeFlexible hoursLittle Wheel
South Plainfield, NJ20 hours ago- ...Hiring!Looking for a part-time job that provides meaningful work and competitive compensation? Consider a position in a State Farm Agent's office.ResponsibilitiesEstablish customer relationships and follow up with customers, as needed.Provide prompt, accurate, and friendly...Part timeFor contractorsWork at officeLocal area
State Farm
Piscataway, NJ4 days ago - ...Overview Looking for a part-time job that provides meaningful work and competitive compensation? Consider a position in a State Farm Agent's office. Responsibilities Establish customer relationships and follow up with customers, as needed. Provide prompt,...Part timeFor contractorsWork at officeLocal area
State Farm
Piscataway, NJ4 days ago $16 - $18 per hour
...skills (written & verbal) Self-motivated with great Organizational skills Computer Savvy Bilingual - Spanish required. As an Agent Team Member, you will receive... Salary + Bonus Health benefits Profit sharing Paid time off (vacation and personal/sick...Hourly payFor contractorsWork at officeState Farm
New Brunswick, NJ20 hours ago- ...Job Description This job is posted on behalf of Sante Consulting LLC by Top Echelon. Position Title: Cytology Manager & FISH Reviewer Department: Cytology FLSA Status: Non-Exempt Position Summary The Cytology/FISH Manager is responsible for microscopically...Daily paidWork at office
Sante Consulting LLC
Edison, NJ2 days ago $60k - $84k
...management activities related to Engineering, Project Management, Supply Chain, Quality, Manufacturing, Customers and suppliers. Review contractual requirements with Project Managers and Project Engineers for data and document delivery prior to and after project...Contract workFor contractorsWorldwideEvapco
Bridgewater, NJ1 day ago$25 - $38 per hour
...Engagement Overview E3n is seeking an independent contractor to support the timely, accurate, and student-centered review of applications for accommodations on the Secondary School Admission Test (SSAT). The engagement is designed to expand review capacity during peak...Hourly payContract workTemporary workFor contractorsSeasonal workRemote workWeekend workAfternoon shiftTalentED Advisors
Skillman, NJ2 days ago- ...Hiring! Looking for a part-time job that provides meaningful work and competitive compensation? Consider a position in a State Farm Agent's office. Responsibilities Establish customer relationships and follow up with customers, as needed. Provide prompt,...Part timeFor contractorsWork at officeLocal area
$23 per hour
...want to consider working in the office of Vipul Verma - State Farm Agent. As a member of our agency team, you have an opportunity to... ...and billing clarification. Use a customer-focused, needs-based review process to educate customers about insurance options. Work with...Hourly payFor contractorsWork at officeLocal areaState Farm
Somerset, NJ4 days ago- ...the DP&C (Group for Document Controls/Information Management and Project Management Procedures and Processes Lead teams to create, review and update Document Controls in accordance with industry and company standards to better support project teams Benchmark and...For contractorsWork at officeImmediate start
- ...Construction). Primary focus will be on procedures and reference manuals. Primary Responsibilities : · Work within a team to develop, update, review and implement processes and procedures with particular focus on 23 DP&C Project Management Procedures and 15 Reference Manuals. ·...
$41.6k - $65k
...Opportunity for advancement Paid time off Training & development Vision insurance Wellness resources ROLE DESCRIPTION Rick Duran - State Farm Agent is a leading insurance agency dedicated to providing exceptional service to our clients. We are currently seeking a dynamic...Full timeFor contractorsWork at office- Documentation Specialist II Provide administrative support to multiple managers within the Contract Development department by submitting requests/entering key information into multiple applications and files, independently manage and maintain trackers, manage mail merge...Contract workWork at office
- ...State Farm Insurance Agent State Farm Insurance Agent located in Somerset, NJ is seeking an outgoing, career-oriented professional to join their team. As a State Farm team member for Vipul Verma - State Farm Agent, you will build and develop customer relationships within...Hourly payFor contractorsLocal area
$18 per hour
...s Circle, Ambassador Travel, and Senior Vice President's Club Apply Now And Let Us Put You On The Path To Success State Farm agents are independent contractors who hire their own employees. State Farm agents' employees are not employees of State Farm. Agents are...Hourly payFor contractorsWork at office
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
