AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$14 per hour
...types of information for top search engines using an online tool.Review and rate search results for relevance and quality, helping to improve... ..., disability, or protected veteran status. At TELUS Digital AI, we are proud to offer equal opportunities and are committed to creating...SuggestedHourly payPart timeFreelanceFlexible hours$97.3k - $125.54k
...JOB DESCRIPTION The position advertised has been exempted from the federal civilian hiring freeze. As an FBI special agent, you'll directly impact national security. By harnessing your background to transition into federal law enforcement, you can help shape the Bureau...SuggestedWork at officeLocal areaFederal Bureau of Investigation (FBI)
Duluth, MN5 days ago- ...Showing Agent Opportunity Do you enjoy being busy? Do you love to work with people and pride yourself on how organized you can be behind the scenes? We may have a place for you as a showing agent on the Heirloom Property Management team! Heirloom is one of the fastest...SuggestedLocal areaMonday to FridayDay shiftAfternoon shift
Heirloom Property Management
Duluth, MN4 days ago - ...relocation industry, is searching for a Right of Way Acquisition Agent III for a project located in Duluth, MN. This position will be... ...title abstracts to determine the parties of interest. Request, review, and evaluate property appraisals or comparable sales reports....SuggestedPermanent employmentFull timeWork experience placementWork at officeRelocation
- Signup Expert is seeking Online Sweepstakes Reviewers & Content Contributors to enhance digital publishing for a leading online sweepstakes review publisher. This fully remote role requires strong written communication and attention to detail but no prior professional experience...SuggestedRemote jobFlexible hours
$15 per hour
...professional office environment. Data Entry Clerk Description Accurately enter credit card information and payment data into company systems Review data for accuracy and completeness before processing Maintain confidential customer and payment information Verify and update...Hourly payTemporary workPart timeWork at officeFlexible hoursShift workDay shiftWeekday work$23.47 - $38.62 per hour
Role Overview USPS is actively accepting applications for Mail Handler Assistants nationwide. This position is open to candidates of all experience levels, with comprehensive training provided to help you succeed. Join a dynamic work environment where you will interact...Hourly payFull timeFlexible hours$17.5 - $19.5 per hour
Job Type: Full-time, Non-exemptAcross Minnesota and Wisconsin, National Bank of Commerce is making a difference in the financial lives of the people, communities, and businesses we serve. We exist to help our customers accomplish more than they could on their own. Join...Hourly payFull timeTemporary workFlexible hoursNATIONAL BANK OF COMMERC
Duluth, MN5 days ago$17.05 - $21.55 per hour
...$21.55 per hour. Grow With Us Your career and earning potential doesn’t stop at hire. Through regular development and performance reviews at 6, 12, 18, and 24 months, high-performing employees may be eligible for pay increases totaling up to 13.5% within the first two...Hourly payFull timeImmediate startRemote workMonday to FridayShift workAffinity Plus Federal Credit Union
Duluth, MN3 days ago$20 per day
We are seeking a compassionate and dedicated Clinical Assistant to join our team in our men's Residential Treatment Program (Howard Friese Prindle). In this role, you will work with a skilled clinical team consisting of Licensed Alcohol & Drug Counselors, Nursing staff...Hourly payFull timeTemporary workPart timeLocal areaShift workNight shiftWeekend workCenter For Alcohol & Drug Treatment
Duluth, MN14 days ago- TellerAre you the kind of person who loves helping others, gets excited about balancing a drawer to the penny, and believes a friendly smile can turn someone's day around? If so, we've got the perfect job for you.We're on the lookout for a Teller who isn't just good with...Local area
Superior Choice Credit Union
Duluth, MN3 days ago - ...orders at the delicatessen counter. Works with co-workers as a team to ensure customer satisfaction and a pleasant work environment. Reviews orders for the day. Understands and practices proper sanitation procedures and ensures the work area is always clean and neat....Work experience placement
- ...Part Time Equal Opportunity Employer This employer is required to notify all applicants of their rights pursuant to federal employment laws. For further information, please review the Know Your Rights notice from the Department of Labor. #J-18808-Ljbffr Miner's Inc.Part timeFlexible hoursShift work
$21.1 - $31.7 per hour
...supporting documentation. Establish and maintain accurate project records to support billing, reporting, and compliance requirements. Review labor, equipment, subcontractor, and material costs to ensure all billable expenses are accurately captured. Prepare and process...Hourly payFull timeContract workTemporary workFor subcontractorWork at officeThe Jamar Company
Duluth, MN3 days ago- ...replenishes or refreshes window washer fluid. Removes trash inside and out in a timely manner. Checks in product, puts product away, and may review invoices. Understands and practices proper sanitation procedures and ensures the work area is always clean and neat. Cleans but not...Night shiftWeekend work
$24 - $30 per hour
Billing Specialist Location: Minneapolis, MN, US, 55113 Duluth, MN, US, 55802 Lake Superior Consulting is seeking a Billing Specialist to join our growing team in either Duluth or Minneapolis, MN! We are flexible with both hybrid and remote work options. This position...Hourly payTemporary workFixed term contractWork at officeRemote workFlexible hoursLake Superior Consulting
Duluth, MN3 days ago- Additional Considerations (if any): Night & Weekend Shifts Required - At Hy-Vee our people are our strength. We promise "a helpful smile in every aisle" and those smiles can only come from a workforce that is fully engaged and committed to supporting our customers and each...Hourly payWork experience placementShift workNight shiftWeekend work
$14 - $16 per hour
...Customer Service Equal Opportunity Employer This employer is required to notify all applicants of their rights pursuant to federal employment laws.For further information, please review the Know Your Rights notice from the Department of Labor. #J-18808-Ljbffr Miner's Inc.Part timeFlexible hoursShift workWeekend workDay shift$45k - $60k
For 100 years, Viking Fire Protection Group has built a reputation for excellence in the design, installation, service, and maintenance of fire protection systems. Our project experience is national, our specialized capabilities are extensive, and we take pride in investing...Work at officeShift work- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
$26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...Hourly payFull timeCurrently hiringWork at office- Marvin is seeking a Product Support Specialist to join our Customer Service & Support team. In this role, you'll assist customers, dealers, and internal partners with technical questions and issues related to our premium windows, doors, and skylights. You will troubleshoot...
Confidential
Duluth, MN15 hours ago $1,200 - $2,500 per week
...performance. Responsibilities: Coordinate freight for owner-operators Communicate with brokers, shippers, and drivers Review and verify load confirmations Track active loads and update statuses Maintain accurate dispatch records Requirements:...For contractorsRemote workAmerican Logistics Authority
Duluth, MN3 days ago$22 - $23 per hour
Sofidel America of Duluth, MN is currently seeking a Shipping-Office Assistant (SOA). We are searching for a dynamic candidate that is local to the area and is looking for a long term role in a stable environment. Your expertise will make you an important part of our team...Work at officeLocal areaMonday to FridayDay shift$50k - $55k
Join VitaCare Living as a Medical Billing Specialist and play a vital role in ensuring our residents receive the compassionate, person-centered care they deserve. You'll be at the heart of our mission, supporting our team by managing billing processes with accuracy and...VitaCare Living
Duluth, MN1 day ago$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for PSE Mail Processing Clerks nationwide. This role is vital to ensuring the timely and accurate processing of mail. USPS provides comprehensive training to help you succeed, even if you have no prior experience. Position...Hourly payPostal Jobs Resource
Duluth, MNmore than 2 months ago$23.47 - $38.62 per hour
Role Overview USPS is accepting applications for Assistant Rural Carriers nationwide. In this role, you deliver and collect packages along designated suburban and rural routes, often using your personal vehicle. The position primarily involves working on weekends and...Hourly payWeekend work$1,200 - $2,500 per week
A logistics company seeks a reliable Entry-Level Freight Dispatcher to support owner-operators with daily dispatch operations. The role offers remote work with estimated weekly earnings between $1,200 and $2,500 based on performance. Responsibilities include coordinating...Remote job- Join VitaCare Living as a Medical Billing Specialist and play a vital role in ensuring our residents receive the compassionate, person-centered care they deserve. You'll be at the heart of our mission, supporting our team by managing billing processes with accuracy and ...
$22 - $36 per hour
The United States Postal Service (USPS) is seeking Mail Carriers and City Carrier Assistants (CCA) to deliver mail and packages to residential and business customers on assigned routes. Mail Carriers are responsible for sorting mail, loading mail and packages into delivery...Hourly payFull timeWork at officeNight shiftUSPS
Duluth, MN5 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
