JSON Specialist - Rubric Expert
$50 - $175 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: JSON-fluent & Rubrics Expert
Type: Contract
Compensation: $50–$175/hour
Location: Remote
Commitment: 20–40 hours/week
Role Responsibilities
- Evaluate JSON-formatted AI model outputs against defined grading rubrics.
- Read and understand written task descriptions and rubrics.
- Parse, read, and assess JSON-based outputs, grading them against provided rubrics.
- Ensure attention to detail and consistency in applying rubric criteria.
- Work independently to meet tight deadlines within 12-24 hours .
Qualifications
Must-Have
- Available to start today, July 24 , running to July 27-28 .
- Comfortable reading and interpreting JSON files .
- Based in the US .
- Access to a desktop or laptop computer (no Chromebooks).
Preferred
- Prior Mercor experience, especially in applying detailed rubrics and evaluation criteria consistently.
Start Date
- Today, July 24
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on ziprecruiter.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the JSON Specialist - Rubric Expert in San Francisco, CA vacancy
$90 - $110 per hour
...investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.Position: BioInformatics Experts Type: Contract Compensation: $90–$110/hour Location: RemoteRole ResponsibilitiesComplete realistic, multi-step scientific data-analysis...SuggestedContract workSummer workMercor Inc
San Francisco, CA1 day ago$70 - $100 per hour
...Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position: Computational Astrophysics & Cosmology Expert Type: Contract Compensation: $70–$100/hour Location: Remote Commitment: 15–20 hours/week Role...SuggestedContract workSummer workRemote workMercor
San Francisco, CAa month ago$24 per hour
...Identify missing assumptions, unclear answers and situations that need clarifying questions. Apply and help refine evaluation rubrics, flag recurring issues and participate in calibration sessions. Collaborate with researchers and other accounting professionals...SuggestedPart timeFreelanceRemote workBraintrust
San Francisco, CA3 days ago$125 - $175 per hour
...Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position: Cybersecurity Practitioner: Paid Expert Interviews (SOC, Incident Response, Detection, AppSec) Type: Contract Compensation: $125–$175/hour Location: Remote...SuggestedHourly payContract workSummer workRemote work$90 per hour
...evaluating AI models. Assess cluster-operations scenarios and manifest correctness. Troubleshoot failure modes and provide clear, rubric-based feedback. Review and improve AI model performance by ensuring production-readiness of tasks. Work independently and...SuggestedRemote jobContract workSummer workMercor
San Francisco, CA9 days ago$100 per hour
...will work as a Healthcare Subject Matter Expert (SME) to create challenging, expert-... ...clinical content, and design high-quality rubrics defining what an accurate, complete, and... ...professional or clinical experience; senior and specialist-level professionals are strongly...Hourly payContract workFor contractorsRemote workSuperAnnotate
San Francisco, CAa month ago- Execute precise tasks in software and document outcomes accurately. About Turing: Turing is one of the world’s fastest-growing AI companies, accelerating the advancement and deployment of powerful AI systems. Turing helps customers in two ways: Working with the...Full timePart timeRemote work
Turing
San Francisco, CA2 days ago $60 - $70 per hour
...Larry Summers , and Jack Dorsey . Position: Fraud Detection Experts Type: Contract Compensation: $60–$70/hour... ...analysis, KYC/AML frameworks, and BSA compliance. Author rubrics that distinguish authentic enterprise fraud investigation judgment...Contract workSummer workRemote workMercor
San Francisco, CA1 day ago$50 per hour
...Paid via ACH, PayPal and AirTM Choose your own schedule. Most experts spend 5-10 hrs/week, up to 40 hours Use GPT, Claude, and... ...advanced generative systems. Key Responsibilites Write Eval Rubrics Debug Agent Traces Stress Test Agent Ideal...Weekly payExtra incomeContract workFreelanceInternshipRemote workWorldwideOutlier
San Francisco, CA3 days ago- ...semantics, and phonology. AfterQuery builds the expert evaluation data that frontier AI labs... ...of research, teaching, or applied work. Specialists in any subfield, syntax, semantics,... ...application gets read against a published rubric and gets a decision. Written assessment....Weekly payContract workTemporary workFor contractorsRemote work10 hours per week
AfterQuery
San Francisco, CAa month ago - ...frontier AI labs use to test model judgment. AfterQuery builds the expert evaluation data that frontier AI labs use to measure and improve... ...Resume review. Every application gets read against a published rubric and gets a decision. Written assessment. You may be asked to...Weekly payContract workTemporary workFor contractorsRemote work10 hours per week
AfterQuery
San Francisco, CAa month ago $25 per hour
...hourly, remote contractor role, you will work as a Persian Language Expert (SME) to review AI-generated Persian content and/or create high-... .../annotation, translation QA, editorial QA, localization QA, or rubric-based evaluation is strongly preferred. Skills: English,...Hourly payTemporary workFor contractorsRemote workShift workSuperAnnotate
San Francisco, CAa month ago$50 - $90 per hour
...enterprise settings. Our AI recruiting agent sources and vets domain experts for our data platform that enables rapid production of high-... ...input for AI training and evaluation. Design conflict-risk rubrics and scenario-based evaluation tools to assess operational and...Part timeFor contractorsRemote workmicro1
San Francisco, CA17 days ago$50 - $80 per hour
...calculations and check analytical solutions. Find and correct errors in AI-generated physical reasoning, units and arithmetic. Write rubrics and reference solutions. Give structured written feedback on model outputs. Follow project guidelines, take part in...Hourly payTemporary workFor contractorsRemote workRemoExperts
San Francisco, CAa month ago$20 per hour
..., remote contractor role, you will work as a Gujarati Language Specialist (SME) to review AI-generated Gujarati content and/or create high... ...AI data training/annotation, translation QA, editorial QA, or rubric-based evaluation is strongly preferred. Skills: Copyediting...Hourly payTemporary workFor contractorsRemote workShift workSuperAnnotate
San Francisco, CAa month ago$50 - $60 per hour
..., and Jack Dorsey . Position: Government & Public Policy Expert Type: Contract Compensation: $50–$60/hour Location... ...reasoning. Provide structured feedback to refine evaluation rubrics and raise quality standards. Qualifications Must-Have ~...Contract workSummer workLocal areaRemote workMercor
San Francisco, CAa month ago$70 - $100 per hour
...Position: Genomics & Cell Biology Specialist Type: Contract Compensation... ...feedback. Develop specifications grading rubrics and verification processes for biology... ...Collaborate with other subject matter experts to ensure consistency and accuracy in...Full timeContract workSummer workRemote workWeekday workMercor
San Francisco, CA21 days ago$130 per hour
...and arriving with an engineering question, positive and negative rubrics, foils, and a golden solution. Explore the repository until... .... Document your work in the task README and submit for expert review, addressing reviewer feedback on returned tasks. Qualifications...Contract workSummer workRemote workMercor
San Francisco, CA21 days ago$100 per hour
...Angelo , Larry Summers , and Jack Dorsey . Position: Office-Suite Experts Type: Contract Compensation: $70-$100/hour Location: Remote... ...task requirements. Grade AI-generated outputs against structured rubrics. Provide written feedback to the research team to improve model...Remote jobContract workSummer workWork at officeImmediate start$60 per hour
...Skill at web research using primary, authoritative sources. Advanced degree holders . Preferred ~ Experience working on rubrics. Application Process (Takes 20–30 mins to complete) Upload resume AI interview based on your resume Submit form Resources...Contract workSummer workRemote workMercor
San Francisco, CA25 days ago$80 - $120 per hour
...Compensation: $80-$120/hour Location: Remote Role Responsibilities Evaluate AI-generated artifacts against domain-specific quality rubrics. Identify factual, aesthetic, and presentation errors in documents, spreadsheets, and slide decks . Provide clear, structured...Contract workSummer workWork at officeRemote work$90 - $100 per hour
...Larry Summers , and Jack Dorsey . Position: Regulatory Law Expert Type: Contract Compensation: $90–$100/hour... ...regulatory filings, and enforcement response documents. Author rubrics that distinguish authentic regulatory judgment from generic law...Contract workSummer workRemote workMercor
San Francisco, CA5 days ago$1,350 per unit
...Help test AI systems on realistic creative and technical workflows. We’re looking for experienced specialists in music, audio, speech and linguistics, or visual design and graphics. What you’ll work on Create practical benchmarks using workflows from your specialty...Braintrust
San Francisco, CA12 hours ago$1,350 per unit
Help evaluate how AI systems approach scientific and clinical work. Use your subject expertise to design realistic problems, define verifiable outcomes and evaluate proposed solutions for scientific accuracy and quality of reasoning. What you’ll work on Develop Linux...Braintrust
San Francisco, CA12 hours ago$50 per hour
...AI projects with leading LLM companies. Pay rate: $50+/hour (depends on role and candidate expertise) Assessment: Shortlisted experts complete an evaluation before selection. Assignments: Contract roles with defined start/end dates; up to 40 hrs/week. About...Contract workRemote workFlexible hoursTuring
San Francisco, CA5 days ago$50 per hour
...projects with leading LLM companies. Pay rate: 50+/hour (depends on role and candidate expertise). Assessment: Shortlisted experts complete an evaluation before selection. Assignments: Contract roles with defined start/end dates; up to 40 hrs/week. About...Contract workRemote workFlexible hoursTuring Inc
San Francisco, CA5 days ago$130k - $180k
...pharmacovigilance professional to serve as our Senior Safety Domain Expert for Veeva Falcon. In this role, you will bridge hands-on case... ...Engagement: Interact directly with PV leads and processing specialists at early adopter organizations, translating their feedback...Work at officeLocal areaWork from homeShift workVeeva Systems
San Francisco, CA1 day ago$48 - $62 per hour
...Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position: AI Safety Experts — English & Swedish Type: Contract Compensation: $48–$62/hour Location: Remote Role Responsibilities Red...Contract workSummer workRemote workMercor Inc
San Francisco, CA2 days ago$60k - $72k
...sleep well so they live well is at the core of what we do. The right mattress matters, and so does the right career. Our Sleep Experts® are at the heart of what we do: helping customers find their dream mattress and get better sleep. By joining our team, you’ll get...Local areaMattress Firm
San Francisco, CA2 days ago$80 - $95 per hour
...Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position: Radio Frequency and Electromagnetic Engineering Expert Type: Contract Compensation: $80–$95/hour Location: Remote Duration: Short-term project Role...Contract workTemporary workSummer workRemote workMercor
San Francisco, CA15 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to JSON Specialist - Rubric Expert. Be the first to apply!
Related searches
- patient safety specialist San Francisco, CA
- credit card specialist San Francisco, CA
- senior specialist San Francisco, CA
- zoning specialist San Francisco, CA
- health program specialist San Francisco, CA
- youth specialist San Francisco, CA
- microsoft specialist San Francisco, CA
- transportation specialist San Francisco, CA
- color specialist San Francisco, CA
- hearing specialist San Francisco, CA