AI Agent Trajectory Annotator and Reviewer
$20 - $30 per hourBespoke Labs
Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
$25 - $35 per hour
Residential Mortgage InspectionsFAR Inspections is actively seeking a reliable independent contractor to perform residential mortgage inspections in Berks County, Pennsylvania and surrounding communities. If you live in or near Reading or anywhere in Berks County, this ...SuggestedHourly payExtra incomeFull timeFor contractorsLive inFlexible hours- ...passion for achieving great things in the world are equally as important to us. Job description Overview The Senior Enterprise AI & Agent Architect will lead the strategy, architecture, governance, and adoption of enterprise AI, Generative AI, and Agentic AI...SuggestedFull timeFlexible hours
$51.63k - $124.72k
...Description Job Description Job Description: Border Patrol Agent (BPA) / Customs and Border Protection Officer (CBPO) About U.S... ...documentation. General Employment Requirements Candidates should review the official announcement for the specific position before...SuggestedFull timeWork experience placement$51.63k - $124.72k
...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law... ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...SuggestedFull timeWork experience placementImmediate startNight shift$51.63k - $124.72k
...Border Patrol Agent (BPA) With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law... ...official employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...SuggestedFull timeWork experience placementImmediate startNight shift$51.63k - $124.72k
...Border Patrol Agent With more than 69,000 employees, U.S. Customs and Border Protection (CBP) is one of the world's largest law enforcement organizations and the largest federal law enforcement agency within the Department of Homeland Security. CBP protects the United...Full timeWork experience placementImmediate startNight shift$10k
...facilitating lawful international travel and trade. As a Border Patrol Agent (BPA) within CBP, you will be a valuable member of the Federal... ...employment site, to complete your application. Be certain to review ALL details within the JOA and follow all instructions in the...Full timeWork experience placementImmediate startNight shift$61k - $65.7k
...one field. This full-time remote job checks assigned changes against the original approved request and records any discrepancy for review. It is office administration that supports better customer service when another colleague relies on the information. You will not infer...Full timeWork at officeRemote workHome office- ...Home, Renters, Life, and other insurance products.Conduct policy reviews to identify coverage gaps and recommend personalized insurance... ...status. This position is with a State Farm independent contractor agent, not with State Farm Insurance Companies. Compensation: $50,000....Base plus commissionFull timeFor contractorsWork at officeLocal areaRelocation package
- Job Description Job Description Job Title: Product Documentation & Labeling Specialist Location: Reading, Pennsylvania Type: Direct Hire Compensation: $20 – $30.00/hr Hours: 8:00 AM – 5:00 PM (flexible) Overview Join a Premium Bakery Brand in the food manufacturing...Work at officeFlexible hours
$20 - $30 per hour
...support the production schedule. Monitor SOP changes that pertain to formula and label revisions and make required updates. Review new ingredients for sub ingredients, no undesirable sub ingredients, trans fat amount, allergens, etc., per regulatory requirements...Hourly payWork at officeLocal areaFlexible hours$60k - $80k
...lead the day-to-day operations of our State Farm agency. This is more than an administrative position. You'll work closely with the agent to keep the office running efficiently, support the team, maintain accountability, improve processes, and ensure our customers receive...For contractorsWork at office- ...increase customer satisfaction and retention. Conduct policy reviews to help ensure customers have the protection they need. Identify... ...fun, challenging, and rewarding, then Seth Walizer - State Farm Agent may be the right fit for you! If you're willing to work hard and...For contractorsWork at office
- ...genetic information, sexual orientation, gender identity, or any other protected status. This position is with a State Farm independent contractor agent, not with State Farm Insurance Companies. Compensation: $50,000.00 - $70,000.00 per year. Seth Walizer - State Farm AgentBase plus commissionFor contractorsLocal area
- Data Entry Operator | Junior (Remote) Reading, PA, United States Job Openings Data Entry Operator | Junior (Remote) A Data Entry Clerk, is responsible for inputting data and making changes to existing data figures in digital databases. Their duties include inputting...Remote job
- ...•Enter, update, and maintain data across Excel, Smartsheet, and internal systems, ensuring information is complete and accurate. •Review and reconcile data across multiple systems and transaction sets, identifying discrepancies and making necessary corrections. •Track...Contract workImmediate startRelocation
$21 per hour
Description Starting Salary: $21 / hour and up As a Citizens Teller - YOU make a real difference for our customers and the branch team. What you'll do You’ll greet our customers in our lobbies. You’re comfortable enough with technology to demonstrate ...Local areaAfternoon shift- Distribution ClerkWork Schedule: 2:00pm-10:30pm, including a weekend and holiday rotation. The Distribution Clerk is part of the moving and receiving team. Performs all work involving the receiving, inspection, loading, unloading, setting up, storing and delivery of supplies...
$750 per week
...time Enjoy free samples from our sponsors and partners in exchange for your honest feedback of their products. You get to review and use new products or services before they are released to the public. You must apply on our website and complete a set of...Extra incomeFull timeTemporary workPart timeLocal areaRemote workWork from homeFlexible hours- Cycle Counter $1.00 INCREASE AFTER 90 DAYS- NEW IMPROVED PAYRATES WITH EXPERIENCE!!!!How You Will Make an Impact As a Cycle counter you will verify the count of quantities, descriptions of products and materials received, shipped, or moved throughout the facility....Local areaShift workDay shift
- To maintain pricing, stocking, and rotation of merchandise in the dairy department. ESSENTIAL JOB FUNCTIONS: 1) Assist in unloading the merchandise. 2) Transport stock for storage to stock areas. 3) Open cartons and price items (if needed) accurately Dairy, Clerk, Retail...
- ...(20%) of the time. Equal Opportunity Employer This employer is required to notify all applicants of their rights pursuant to federal employment laws. For further information, please review the Know Your Rights notice from the Department of Labor. Redner's MarketsPart timeLocal area
- ...daily settlement and proof of cash transactions; balance cash drawer accurately, quickly and efficiently on a daily basis.Prepare and review reports relating to the function, e.g. Currency Transaction Reports (CTRs), Bank Secrecy Reports, etc.Process night deposit, mail...Work at officeLocal areaNight shiftAfternoon shift
$17.25 - $24.23 per hour
Job TitleThis is a full-time career opportunity (approximately 40 hours per week) based out of our Muhlenberg financial center in Reading, PA. Must be available to be scheduled around branch operation hours (no Sundays).Responsible for providing an exceptional Financial...Hourly payFull timeTemporary workWork experience placementWork visaFlexible hours$26.47 - $43.62 per hour
The Postal Service is the largest government-related agency, employing nearly 1 million people, and is currently hiring nationwide with many positions available. Job Description Clerk – Clerks sell stamps, take incoming mail, and sort and distribute mail in addition...Hourly payFull timeCurrently hiringWork at office$26.47 - $43.62 per hour
...answers – To make sure you are fully prepared, you will be given sample exam questions AND answers. You will be shown exactly what to review and how to review, so you can get the highest score possible ~ Salary Gradings – You will be given a list of salary gradings for...Hourly payFull timeCurrently hiringImmediate startFlexible hours- ...verbal instructions. Equal Opportunity Employer This employer is required to notify all applicants of their rights pursuant to federal employment laws. For further information, please review the Know Your Rights notice from the Department of Labor. Redner's MarketsShift workNight shift
- ...Service department. Equal Opportunity Employer This employer is required to notify all applicants of their rights pursuant to federal employment laws. For further information, please review the Know Your Rights notice from the Department of Labor. Redner's MarketsLocal area
- Dairy Clerk To maintain pricing, stocking, and rotation of merchandise in the dairy department. Assist in unloading the merchandise. Transport stock for storage to stock areas. Open cartons and price items (if needed) accurately. Keep store shelves fully...Shift work
$13 per hour
Frozen Foods Department Assistant To maintain pricing, stocking, and rotation of merchandise in the frozen foods department to ensure freshness and availability of products. Assist in unloading the merchandise when the deliveries arrive at the store. Transport stock...Shift work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Trajectory Annotator and Reviewer. Be the first to apply!





