JSON Specialist - Rubric Expert
$50 - $175 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: JSON-fluent & Rubrics Expert
Type: Contract
Compensation: $50–$175/hour
Location: Remote
Commitment: 20–40 hours/week
Role Responsibilities
- Evaluate JSON-formatted AI model outputs against defined grading rubrics.
- Read and understand written task descriptions and rubrics.
- Parse, read, and assess JSON-based outputs, grading them against provided rubrics.
- Ensure attention to detail and consistency in applying rubric criteria.
- Work independently to meet tight deadlines within 12-24 hours .
Qualifications
Must-Have
- Available to start today, July 24 , running to July 27-28 .
- Comfortable reading and interpreting JSON files .
- Based in the US .
- Access to a desktop or laptop computer (no Chromebooks).
Preferred
- Prior Mercor experience, especially in applying detailed rubrics and evaluation criteria consistently.
Start Date
- Today, July 24
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on ziprecruiter.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the JSON Specialist - Rubric Expert in San Francisco, CA vacancy
$90 - $110 per hour
...investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.Position: BioInformatics Experts Type: Contract Compensation: $90–$110/hour Location: RemoteRole ResponsibilitiesComplete realistic, multi-step scientific data-analysis...SuggestedContract workSummer workMercor Inc
San Francisco, CA4 days ago$70 - $100 per hour
...Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position: Computational Astrophysics & Cosmology Expert Type: Contract Compensation: $70–$100/hour Location: Remote Commitment: 15–20 hours/week Role...SuggestedContract workSummer workRemote workMercor
San Francisco, CAa month ago$60 - $80 per hour
...Compensation: $60$80/hour Location: Remote Role Responsibilities Review expert-written tasks including prompts golden outputs and rubrics. Evaluate the accuracy and depth of AI-generated content to strengthen reasoning and rigor in...SuggestedFull timeContract workSummer workRemote work$24 per hour
...Identify missing assumptions, unclear answers and situations that need clarifying questions. Apply and help refine evaluation rubrics, flag recurring issues and participate in calibration sessions. Collaborate with researchers and other accounting professionals...SuggestedPart timeFreelanceRemote workBraintrust
San Francisco, CA1 day ago$125 - $175 per hour
...General Catalyst Peter Thiel Adam DAngelo Larry Summers and Jack Dorsey . Position: Cybersecurity Practitioner: Paid Expert Interviews (SOC Incident Response Detection AppSec) Type: Contract Compensation: $125$175/hour Location:...SuggestedHourly payFull timeContract workSummer workRemote work$100 per hour
...will work as a Healthcare Subject Matter Expert (SME) to create challenging, expert-... ...clinical content, and design high-quality rubrics defining what an accurate, complete, and... ...professional or clinical experience; senior and specialist-level professionals are strongly...Hourly payContract workFor contractorsRemote workSuperAnnotate
San Francisco, CAa month ago- Execute precise tasks in software and document outcomes accurately. About Turing: Turing is one of the world’s fastest-growing AI companies, accelerating the advancement and deployment of powerful AI systems. Turing helps customers in two ways: Working with the...Full timePart timeRemote work
Turing
San Francisco, CA3 hours ago $90 per hour
...evaluating AI models. Assess cluster-operations scenarios and manifest correctness. Troubleshoot failure modes and provide clear, rubric-based feedback. Review and improve AI model performance by ensuring production-readiness of tasks. Work independently and...Remote jobContract workSummer workMercor
San Francisco, CA7 days ago$50 per hour
...Paid via ACH, PayPal and AirTM Choose your own schedule. Most experts spend 5-10 hrs/week, up to 40 hours Use GPT, Claude, and... ...advanced generative systems. Key Responsibilites Write Eval Rubrics Debug Agent Traces Stress Test Agent Ideal...Weekly payExtra incomeContract workFreelanceInternshipRemote workWorldwideOutlier
San Francisco, CA1 day ago- ...semantics, and phonology. AfterQuery builds the expert evaluation data that frontier AI labs... ...of research, teaching, or applied work. Specialists in any subfield, syntax, semantics,... ...application gets read against a published rubric and gets a decision. Written assessment....Weekly payContract workTemporary workFor contractorsRemote work10 hours per week
AfterQuery
San Francisco, CA29 days ago $90 - $100 per hour
...Larry Summers , and Jack Dorsey . Position: Regulatory Law Expert Type: Contract Compensation: $90–$100/hour... ...regulatory filings, and enforcement response documents. Author rubrics that distinguish authentic regulatory judgment from generic law...Contract workSummer workRemote workMercor
San Francisco, CAa month ago$800 per month
...Jack Dorsey . Position: Reinsurance & Program Business Specialist Type: Contract Compensation: $800/hour Location:... ...quality. Grade AI-generated responses against structured rubrics for structural accuracy, contract logic, financial understanding...Contract workSummer workImmediate startRemote workMercor
San Francisco, CA9 days ago$100 per hour
...Angelo , Larry Summers , and Jack Dorsey . Position: Office-Suite Experts Type: Contract Compensation: $70-$100/hour Location: Remote... ...task requirements. Grade AI-generated outputs against structured rubrics. Provide written feedback to the research team to improve model...Remote jobContract workSummer workWork at officeImmediate start- ...As a legal expert, you'll help train and evaluate how AI models reason about the law. This can be writing challenging legal prompts,... ...like fabricated citations or misapplied rules, drafting model answers and explanations, or creating rubrics to grade response quality....Temporary work
AfterQuery
San Francisco, CA29 days ago - ...frontier AI labs use to test model judgment. AfterQuery builds the expert evaluation data that frontier AI labs use to measure and improve... ...Resume review. Every application gets read against a published rubric and gets a decision. Written assessment. You may be asked to...Weekly payContract workTemporary workFor contractorsRemote work10 hours per week
AfterQuery
San Francisco, CA29 days ago $25 per hour
...hourly, remote contractor role, you will work as a Persian Language Expert (SME) to review AI-generated Persian content and/or create high-... .../annotation, translation QA, editorial QA, localization QA, or rubric-based evaluation is strongly preferred. Skills: English,...Hourly payTemporary workFor contractorsRemote workShift workSuperAnnotate
San Francisco, CA29 days ago$50 - $90 per hour
...enterprise settings. Our AI recruiting agent sources and vets domain experts for our data platform that enables rapid production of high-... ...input for AI training and evaluation. Design conflict-risk rubrics and scenario-based evaluation tools to assess operational and...Part timeFor contractorsRemote workmicro1
San Francisco, CA15 days ago$100 per hour
...Role Overview Turing is looking for Portfolio Management experts to work with our researchers to improve the performance of AI models... ...return analysis, and portfolio optimization. Create rubrics to assess model capabilities on tasks like performance attribution...Hourly payPart timeFreelanceRemote workFlexible hoursTuring
San Francisco, CA6 days ago$60 - $70 per hour
...Larry Summers , and Jack Dorsey . Position: Fraud Detection Experts Type: Contract Compensation: $60–$70/hour... ...analysis, KYC/AML frameworks, and BSA compliance. Author rubrics that distinguish authentic enterprise fraud investigation judgment...Contract workSummer workRemote workMercor
San Francisco, CAa month ago$90 - $150 per hour
...seeking talented licensed Clinicians and Health subject-matter experts (SMEs) with clinical expertise and active licensure in medicine... ...produce accurate, guideline-based reference answers. Develop rubrics to assess clinical accuracy, safety and appropriateness of AI-...Contract workPart timeWeekday workMercor
San Francisco, CA3 days ago$15 per hour
...hourly, remote contractor role, you will work as a Tamil Language Specialist (SME) to review AI-generated Tamil content and/or create high-... ...AI data training/annotation, translation QA, editorial QA, or rubric-based evaluation is strongly preferred. Skills:...Hourly payTemporary workFor contractorsRemote workShift workSuperAnnotate
San Francisco, CA29 days ago$70 - $100 per hour
...Position: Genomics & Cell Biology Specialist Type: Contract Compensation... ...feedback. Develop specifications grading rubrics and verification processes for biology... ...Collaborate with other subject matter experts to ensure consistency and accuracy in...Full timeContract workSummer workRemote workWeekday workMercor
San Francisco, CA19 days ago$50 - $80 per hour
...calculations and check analytical solutions. Find and correct errors in AI-generated physical reasoning, units and arithmetic. Write rubrics and reference solutions. Give structured written feedback on model outputs. Follow project guidelines, take part in...Hourly payTemporary workFor contractorsRemote workRemoExperts
San Francisco, CA29 days ago$130 per hour
...and arriving with an engineering question, positive and negative rubrics, foils, and a golden solution. Explore the repository until... .... Document your work in the task README and submit for expert review, addressing reviewer feedback on returned tasks. Qualifications...Contract workSummer workRemote workMercor
San Francisco, CA19 days ago- ...real trade work. Remote, async, paid weekly. AfterQuery builds the expert evaluation data that frontier AI labs use to measure and improve... ...Resume review. Every application gets read against a published rubric and gets a decision. Written assessment. You may be asked to...Weekly payContract workTemporary workFor contractorsApprenticeshipRemote work10 hours per week
AfterQuery
San Francisco, CA29 days ago - ...AI labs use to test veterinary reasoning. AfterQuery builds the expert evaluation data that frontier AI labs use to measure and improve... ...Resume review. Every application gets read against a published rubric and gets a decision. Written assessment. You may be asked to take...Weekly payContract workTemporary workFor contractorsRemote work10 hours per week
AfterQuery
San Francisco, CA29 days ago $50 - $55 per hour
...training data quality and downstream performance. Develop evaluation rubrics and benchmarks for assessing AI-generated sales tasks and solutions. Collaborate with subject matter experts to ensure consistency, relevance, and coverage across datasets. Work...Full timeContract workSummer workRemote workMercor
San Francisco, CA2 days ago$70 - $90 per hour
...tasks. Assess multi-service serverless designs , IaC fidelity , and cross-service integration correctness. Provide clear, rubric-based written feedback to improve AI model training . Review event-driven architecture for correctness, including IAM/...Contract workSummer workLocal areaRemote workMercor
San Francisco, CA22 days ago$60 per hour
...Skill at web research using primary, authoritative sources. Advanced degree holders . Preferred ~ Experience working on rubrics. Application Process (Takes 20–30 mins to complete) Upload resume AI interview based on your resume Submit form Resources...Contract workSummer workRemote workMercor
San Francisco, CA23 days ago$1,350 per unit
Help evaluate how AI systems approach scientific and clinical work. Use your subject expertise to design realistic problems, define verifiable outcomes and evaluate proposed solutions for scientific accuracy and quality of reasoning. What you’ll work on Develop Linux...Braintrust
San Francisco, CA3 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to JSON Specialist - Rubric Expert. Be the first to apply!
Related searches
- patient safety specialist San Francisco, CA
- credit card specialist San Francisco, CA
- senior specialist San Francisco, CA
- zoning specialist San Francisco, CA
- health program specialist San Francisco, CA
- microsoft specialist San Francisco, CA
- transportation specialist San Francisco, CA
- color specialist San Francisco, CA
- hearing specialist San Francisco, CA
- crisis specialist San Francisco, CA