AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$41.6k
...timeliness, accuracy and productivity standards. Documents quality issues, performance measures and quality improvements for management review. Provides information to assist in giving feedback to employees and in the training of employees as needed. Ensures proper...SuggestedFull timeTemporary workLocal area3 days per week$34.57 - $46.11 per hour
...every stage of your career. Try new things, learn new skills and discover what you excel at—all from Day One. Job Description Reviews appraisals and reports including desktop valuations from a wide variety of bank channels. Reviewer Appraiser positions review all types...SuggestedTemporary workWork at officeLocal area3 days per weekU.S. Bank
Owensboro, KY1 day ago$18 - $25 per hour
Job Description Job Description Document Controller (Light Drafting) Company: Dynamic Steel LLC Location: Owensboro, KY (On-site) Job Type: Full-time Pay: $18-$25/hr, depending on experience About the Role Dynamic Steel LLC is hiring a Document Controller...SuggestedFull timeWork at officeDynamic Steel LLC
Owensboro, KY4 days ago$20 per hour
...timeliness, accuracy and productivity standards. Documents quality issues, performance measures and quality improvements for management review. Provides information to assist in giving feedback to employees and in the training of employees as needed. Ensures proper...SuggestedTemporary work3 days per week- ...Shelter Insurance Agent Opportunity A company built to serve you. It's your career, Shelter it! We are seeking an exceptional entrepreneur to make a difference in Owensboro, KY and become our newest Shelter Insurance Agent. Do you have a desire to meet new people...SuggestedContract workWork at officeRelocation
- Loan Documentation Specialist The Loan Documentation Specialist - Pre Close will support our lending staff in preparing and processing essential loan documents for many types of loans, including retail, commercial, agricultural, and home equity lines of credit. Essential...Part timeWork at office
- Data Entry Operator | Junior (Remote)Owensboro, KY, United States Job Openings Data Entry Operator | Junior (Remote)About the Job Data Entry Operator | Junior (Remote)Important: You will receive an email within the next two minutes after applying, check your inbox or spam...Remote work
Only Data Entry
Owensboro, KY2 days ago - About the job Data Entry Operator | Junior (Remote) Important: You Will Receive An Email Within Next 2 Minutes After Applying , Check Your Inbox or Spam Folder For next steps. A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures...Remote job
- Job Description Job Description Benefits: ~401(k) matching ~ Competitive salary ~ Opportunity for advancement ~ Training & development You May Be a Great Fit as an Acquisitions Specialist at Seth Lawson State Farm if: You’re the person people call first...Work at office
$750 per week
ApexFocusGroup collaborates with research firms, universities, and trusted brands to gather genuine consumer insights. We help connect everyday people with flexible remote market research opportunities that influence real products, services, and customer experiences. Role...Part timeCasual workRemote workFlexible hoursAFG
Owensboro, KY4 days ago- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
$26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...Hourly payFull timeCurrently hiringWork at office- ...they want and need. Adequately prepare, package, label and inventory merchandise. Check product quality to ensure freshness. Review "sell by" dates and take appropriate action. Label, stock and inventory department merchandise. Report product ordering/shipping...Seasonal work
- *Position is not eligible for remote work. Job Code TELLI Job Family Supervisory Job Summary A Teller undertakes bank transactions and interacts directly with customers. A Teller is usually the first person a customer sees when they walk into one of our locations. This ...Work at office
Independence Bank
Owensboro, KY4 days ago - Independence Bank in Owensboro, KY seeks a Teller to handle daily bank transactions and interact with customers. The role includes working Saturday rotations and potential drive-thru duties, with opportunities to promote banking products. The ideal candidate has 1 year...
Independence Bank
Owensboro, KY2 days ago - Premier Fire & Security, an APi Group Life Safety USA company, has an opening for a self-motivated individual to join our team as an Office Administrator . This position will work from our Owensboro, Kentucky office. Premier Fire & Security has over 50 years of experience...Work at office
$1,500 - $3,000 per week
Earn $1,500–$3,000+ Weekly | Now Hiring Freight Dispatchers (Experienced & Entry-Level) Truck Driver Nation is growing, and we’re looking for Freight Dispatchers ready to take control of their financial future. Experienced Dispatchers – Put your skills to work with...Immediate startAmerican Logistics Authority
Owensboro, KY4 days ago$2,200 - $3,000 per week
A logistics company is seeking motivated individuals for the remote role of Freight Dispatcher. This position offers a strong income potential, with earnings ranging from $2,200 to $3,000 weekly. Candidates should be confident in negotiating rates, familiar with dispatching...Remote workAmerican Logistics Authority
Owensboro, KY2 days ago$1,800 - $3,200 per week
Entry-Level Freight Dispatcher – Weekly Pay: $1,800 to $3,200+ Job ID: ALA-D1A We are currently hiring motivated and detail-oriented individuals to join our team as Independent Freight Dispatchers. What “Entry-Level” Means: If you already have basic freight...Weekly payFor contractorsCurrently hiringFlexible hoursAmerican Logistics Authority
Owensboro, KY5 days ago$1,800 - $4,500 per week
Freight Dispatcher (Independent Contractor) — $1,800 to $4,500+ Weekly Potential Employment Type: 1099 / Independent Contractor Location: Remote (U.S. Based Only) This is a performance-based dispatch role working with owner-operators under their own authority...For contractorsRemote workAmerican Logistics Authority
Owensboro, KY4 days ago- A logistics solutions provider is seeking a Freight Dispatcher (Independent Contractor) with high earning potential, working remotely in the U.S. This performance-based role involves booking loads, negotiating rates, and coordinating logistics. Candidates should be organized...For contractorsRemote work
American Logistics Authority
Owensboro, KY2 days ago $2,200 - $3,000 per week
Entry-Level Freight Dispatcher (Remote) – Earn $2,200 to $3,000 Weekly ($114,400–$156,000 Annually) We are hiring motivated individuals to join our logistics team as Freight Dispatchers. This is a remote position with strong income potential. Candidates must already ...Weekly payStart working todayWork at officeImmediate startRemote workWork from homeHome officeAmerican Logistics Authority
Owensboro, KY3 days ago- Weekend Requirements: N/A | Work Hours: M-F 8-430, M-F 730-4 | Status: Full Time - 1.0 FTE (80 hours per pay period) | Entity: Owensboro Health Regional Hospital | Department: Wound Healing Owensboro Job Summary: Responsible for a variety of clinical and administrative...Full timeWork experience placementWork at officeShift workWeekend work
Owensboro Health
Owensboro, KY5 days ago $1,500 - $3,000 per week
A leading logistics firm is seeking Freight Dispatchers in Owensboro, Kentucky. The role offers independent work with potential earnings between $1,500 and $3,000 weekly, depending on performance. Candidates will manage multiple trucks and need strong communication and...American Logistics Authority
Owensboro, KY2 days ago$6,000 per month
Freight Dispatchers Wanted Truck Driver Nation is expanding and we are hiring both experienced and entry-level freight dispatchers who are ready to take charge of their financial future. This is not an hourly job. As an independent freight dispatcher, you earn 8%...Hourly payImmediate startAmerican Logistics Authority
Owensboro, KY5 days ago$20 - $36 per hour
Medical Coder Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and ...Hourly payMinimum wageFull timeWork experience placementLocal areaRemote workMonday to FridayShift work- ...we can't wait to meet you! Equal Opportunity Employer This employer is required to notify all applicants of their rights pursuant to federal employment laws. For further information, please review the Know Your Rights notice from the Department of Labor. DeaconessWork at office
$8,000 per month
Freight Dispatchers Wanted (Experienced & Entry-Level) - Work From Home & Earn $8,000 to $15,000+ Per Month Truck Driver Nation is hiring motivated individuals ready to build a high-paying career from the comfort of home. Whether you’re an experienced dispatcher or brand...Hourly payWork from home- NO EXPERIENCE REQUIRED – PAID TRAINING PROVIDED – JOB SECURITY The Postal Service is the largest government related agency in terms of employees. The Postal Service currently employs nearly 1 million people and is hiring nationwide. There is NO experience required, ...Full time
$22 - $36 per hour
The United States Postal Service (USPS) is seeking Mail Carriers and City Carrier Assistants (CCA) to deliver mail and packages to residential and business customers on assigned routes. Mail Carriers are responsible for sorting mail, loading mail and packages into delivery...Hourly payFull timeWork at officeNight shiftUSPS
Owensboro, KY2 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!
