JSON Evaluator - Fully Remote | Upto $175/hr
$175 per hourMercor
San Francisco, CA
- Remote job
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: JSON-fluent & Rubrics Expert
Type: Contract
Compensation: $50–$175/hour
Location: Remote
Commitment: 20–40 hours/week
Role Responsibilities
- Evaluate JSON-formatted AI model outputs against defined grading rubrics.
- Read and understand written task descriptions and rubrics.
- Parse, read, and assess JSON-based outputs, grading them against provided rubrics.
- Ensure attention to detail and consistency in applying rubric criteria.
- Work independently to meet tight deadlines within 12-24 hours .
Qualifications
Must-Have
- Available to start today, July 24 , running to July 27-28 .
- Comfortable reading and interpreting JSON files .
- Based in the US .
- Access to a desktop or laptop computer (no Chromebooks).
Preferred
- Prior Mercor experience, especially in applying detailed rubrics and evaluation criteria consistently.
Start Date
- Today, July 24
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on ziprecruiter.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 11 days ago
Similar jobs that could be interesting for youBased on the JSON Evaluator - Fully Remote | Upto $175/hr in San Francisco, CA vacancy
$19 per hour
...Expert - Bengali Type: Contract Compensation: $11–$19/hour Location: Remote Duration: Up to 6 months Commitment: 20 hours/week Role Responsibilities Evaluate AI-generated music across diverse genres and rate against quality standards....Remote jobContract workSummer workImmediate startMercor
San Francisco, CA18 days ago$120 per hour
..., Larry Summers , and Jack Dorsey . Position: Media / journalism / communications Evaluator Type: Contract Compensation: $80–$120/hour Location: Remote Role Responsibilities Evaluate AI-generated artifacts against domain-specific quality...Remote jobContract workSummer workWork at officeMercor
San Francisco, CA13 days ago$120 per hour
..., Larry Summers , and Jack Dorsey . Position: Real estate / hospitality / events Evaluator Type: Contract Compensation: $80–$120/hour Location: Remote Role Responsibilities Evaluate AI-generated artifacts against domain-specific quality...Remote jobContract workSummer workWork at officeMercor
San Francisco, CAa month ago$90 per hour
.... Position: ML Challenge Task Auditor Type: Contract Compensation: $70–$90/hour Location: Remote Role Responsibilities Evaluate the quality, correctness, and methodological rigor of applied machine-learning tasks for training and evaluating...Remote jobContract workSummer workMercor
San Francisco, CA15 days ago$17 per hour
...Audio Expert - Thai Type: Contract Compensation: $17/hour Location: Remote Duration: Up to 6 months Commitment: 10+ hours/week Role Responsibilities Evaluate AI model output lyrics , voice generation, and other standards in various music...Remote jobContract workSummer workImmediate startMercor
San Francisco, CA15 days ago$90 per hour
...Dorsey . Position: SWE-Bench Task Auditor Type: Contract Compensation: $70–$90/hour Location: Remote Role Responsibilities Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks . Assess repository-...Remote jobContract workSummer workMercor
San Francisco, CA15 days ago$90 per hour
...Position: AI Developer Trace Task Auditor Type: Contract Compensation: $70–$90/hour Location: Remote Role Responsibilities Evaluate the quality and correctness of AI-assisted software-development traces to enhance model training and evaluation...Remote jobContract workSummer workMercor
New York, NY15 days ago$42 per hour
...Expert - Urdu Type: Contract Compensation: $14–$42/hour Location: Remote Duration: Up to 6 months Commitment: 20+ hours/week Role Responsibilities Evaluate AI-generated music across various genres and rate it against quality standards....Remote jobContract workSummer workImmediate startMercor
New York, NY4 days ago$120 per hour
...Angelo , Larry Summers , and Jack Dorsey . Position: Clinical / biomedical / pharma Evaluator Type: Contract Compensation: $80–$120/hour Location: Remote Role Responsibilities Evaluate AI-generated artifacts against domain-specific quality...Remote jobContract workSummer workWork at officeMercor
New York, NY21 days ago$42 per hour
...Expert - Hebrew Type: Contract Compensation: $17–$42/hour Location: Remote Duration: Up to 6 months Commitment: 20+ hours/week Role Responsibilities Evaluate AI-generated music across various genres for quality and creativity. Compare...Remote jobContract workSummer workImmediate startMercor
New York, NYa month ago$160 per hour
...: Contract Compensation: $80–$160/hour Location: Remote Role Responsibilities Review and assess domain-specific documents... ...Provide structured feedback to improve document standards and evaluation processes. Utilize Excel , PowerPoint , and Word for...Remote jobContract workSummer workMercor
New York, NY29 days ago$190 per hour
...Plug In Connectors Expert Type: Contract Compensation: $50–$190/hour Location: Remote Commitment: 20+ hours/week Role Responsibilities Evaluate AI systems on complex personal workflows, including personal health , travel , and activity...Remote jobHourly payContract workFor contractorsSummer workTrial periodMercor
New York, NY14 days ago$175 per hour
...Larry Summers , and Jack Dorsey . Position: CNC Machining Expert Type: Contract Compensation: $120–$175/hour Location: Remote Role Responsibilities Contribute to text-only, objective, verifiable rubrics for grading CNC machining solutions...Remote jobContract workPart timeSummer workMercor
New York, NY3 days ago$175 per hour
...Jack Dorsey . Position: Epidemiologist — Patient Population Sizing Type: Contract Compensation: $150–$175/hour Location: Remote Duration: 1–2 week pilot Commitment: 10–20 hours Role Responsibilities Create and critique rubrics that...Remote jobHourly payContract workPart timeFor contractorsSummer workMercor
New York, NY5 days ago$175 per hour
...Contractor \n \n Compensation: $175–$300/hour \n \n Location: Remote \n \n About the Opportunity... ..., financial modeling, earnings evaluation, valuation, and investment research... ...n Additional Details \n \n # Fully remote opportunity. \n #...Remote jobFor contractorsImmediate startFlexible hoursSynthires
West Virginia6 days ago$70 per hour
...Dorsey . Position: Project Coordinator - AI & Data Projects Type: Contract Compensation: $45–$70/hour Location: Remote Role Responsibilities Act as a day-to-day point of contact for AI data projects . Ensure workflows and operations run...Remote jobContract workSummer workFlexible hoursMercor
San Francisco, CA15 days ago$50 per hour
...Type: Contract Compensation: $20–$50/hour Location: Remote Role Responsibilities Review work product against defined... ...Help refine and improve quality processes, checklists, and evaluation criteria. Collaborate asynchronously with a distributed team...Remote jobContract workSummer workMercor
New York, NY23 days ago$80 - $120 per hour
...$120 per hour We are hiring expert Evaluators in Special education / IEP to review and... ...expertise to grade outputs. This is a remote, hourly engagement. Requirements... ...an independent contractor. This is a fully remote role that can be completed on your...Remote workHourly payContract workFor contractorsWork at officeWeekday AI
United States4 days ago$37.5 per hour
...In this role you will be responsible for evaluating advertising copy generated by a large... ...range for this position is $37.50 - $37.50/hr. Individual compensation offered for... ...Vacation or Sick Leave)Workplace TypeThis is a fully remote position.Application DeadlineThis...Remote workContract workTemporary workTEKsystems
New York, NY4 days ago$100 per hour
..., and Jack Dorsey . Position: Business Domain Expert — AI Model Evaluation Pilot (Admin, Marketing, HR, Accounting) Type: Contract Compensation: $70–$100/hour Location: Remote Role Responsibilities Solve one realistic, domain-specific task using...Remote jobContract workSummer workMercor
New York, NY25 days ago$150 per hour
...Contract Compensation: $100–$150/hour Location: Remote Role Responsibilities Design precise, task-specific grading... ...judgment. Preferred ~ Prior experience with AI training , evaluation, or human-data projects. Application Process (Takes 20–30...Remote jobContract workSummer workMercor
New York, NY14 days ago$100 per hour
...: Contract Compensation: $60–$100/hour Location: Remote Role Responsibilities Design task briefs based on actual... ...plausible-but-wrong responses. Review AI agent attempts and evaluate them against your standards. Diagnose issues from messy or...Remote jobContract workSummer workMercor
New York, NY27 days ago$100 per hour
...: Contract Compensation: $60–$100/hour Location: Remote Role Responsibilities Design task briefs from real brand... ...Review AI agent attempts. Judge them against your standards. Evaluate AI capabilities in diagnosing data, prioritizing, planning, QA,...Remote jobContract workSummer workMercor
New York, NY27 days ago$80 per hour
..., and Jack Dorsey . Position: Business Domain Expert — AI Model Evaluation Pilot (Admin, Marketing, HR, Accounting) Type: Contract Compensation: $60–$80/hour Location: Remote Role Responsibilities Sign an NDA before receiving any materials....Remote jobHourly payContract workFor contractorsSummer workMercor
New York, NY3 days ago$160 per hour
...: Contract Compensation: $80–$160/hour Location: Remote Role Responsibilities Review and assess domain-specific documents... ...Utilize strong analytical skills to contribute to high-impact evaluation projects. Work independently and asynchronously to meet...Remote jobContract workSummer workMercor
New York, NYa month ago$160 per hour
...: Contract Compensation: $80–$160/hour Location: Remote Role Responsibilities Review and assess domain-specific documents... ...impact. Utilize strong analytical skills to contribute to evaluation projects. Collaborate with teams to enhance document...Remote jobContract workSummer workMercor
New York, NY29 days ago$100 per hour
...: Contract Compensation: $60–$100/hour Location: Remote Role Responsibilities Design task briefs from real marketing... ...attempts and judge them against established standards. Evaluate AI-generated content in Marketing Analytics to improve model...Remote jobContract workSummer workMercor
New York, NY19 days ago- ...is seeking a Population Health Informaticist (AI Training) to evaluate AI-generated population health analyses and data pipelines. You... ...AI reasoning on population health concepts. The role is fully remote, hourly contract, and offers flexible scheduling (10–40 hours/...Remote workHourly payContract workFlexible hours
Alignerr Corp.
Raleigh, NC1 hour ago $80 per hour
...Type: Contract Compensation: $60–$80/hour Location: Remote Role Responsibilities Guide research and engineering... ...well-reasoned solutions grounded in real marketing practice. Evaluate AI model outputs against structured rubrics and provide clear...Remote jobContract workPart timeSummer workWeekday workMercor
New York, NY16 days ago$90 per hour
...Type: Contract Compensation: $90/hour Location: Remote Duration: September 12–15, 2026 Commitment: 13 hours... ...Work independently to meet deadlines and ensure high-quality evaluations. Qualifications Must-Have ~5+ years of experience in...Remote jobHourly payContract workFor contractorsSummer workMercor
New York, NY11 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to JSON Evaluator - Fully Remote | Upto $175/hr. Be the first to apply!
Related searches
- quality evaluator San Francisco, CA
- evaluator San Francisco, CA
- social media evaluator San Francisco, CA
- education evaluator San Francisco, CA
- ai evaluator San Francisco, CA
- work from home web search evaluator San Francisco, CA
- software engineering manager remote San Francisco, CA
- remote recruitment consultant San Francisco, CA
- remote data manager San Francisco, CA
- senior php developer remote San Francisco, CA