JSON Specialist - Rubric Expert
$50 - $175 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: JSON-fluent & Rubrics Expert
Type: Contract
Compensation: $50–$175/hour
Location: Remote
Commitment: 20–40 hours/week
Role Responsibilities
- Evaluate JSON-formatted AI model outputs against defined grading rubrics.
- Read and understand written task descriptions and rubrics.
- Parse, read, and assess JSON-based outputs, grading them against provided rubrics.
- Ensure attention to detail and consistency in applying rubric criteria.
- Work independently to meet tight deadlines within 12-24 hours .
Qualifications
Must-Have
- Available to start today, July 24 , running to July 27-28 .
- Comfortable reading and interpreting JSON files .
- Based in the US .
- Access to a desktop or laptop computer (no Chromebooks).
Preferred
- Prior Mercor experience, especially in applying detailed rubrics and evaluation criteria consistently.
Start Date
- Today, July 24
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on ziprecruiter.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 11 days ago
Similar jobs that could be interesting for youBased on the JSON Specialist - Rubric Expert in San Francisco, CA vacancy
$20 per hour
...What You'll Do Call prospective experts to discuss paid research and AI training opportunities Explain project scopes, answer questions, and gauge candidate interest Guide qualified candidates through the initial onboarding steps Track call outcomes and maintain accurate...SuggestedContract workTemporary workFor contractorsRemote workTerac
San Francisco, CA7 days ago$70 - $100 per hour
...Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position: Computational Astrophysics & Cosmology Expert Type: Contract Compensation: $70–$100/hour Location: Remote Commitment: 15–20 hours/week Role...SuggestedContract workSummer workRemote workMercor
San Francisco, CA5 days ago$90 - $110 per hour
...Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position: BioInformatics Experts Type: Contract Compensation: $90–$110/hour Location: Remote Role Responsibilities Complete realistic, multi-...SuggestedContract workSummer workRemote workMercor
San Francisco, CA10 days ago$75 per hour
...Larry Summers , and Jack Dorsey . Position: Architecture Expert Type: Contract Compensation: $75/hour Location:... ..., grounded tasks with clear ground-truth outputs and objective rubrics. Review and improve AI model outputs for accuracy and alignment...SuggestedContract workSummer workRemote workMercor
San Francisco, CA5 days ago$70 - $90 per hour
...tasks. Assess multi-service serverless designs , IaC fidelity , and cross-service integration correctness. Provide clear, rubric-based written feedback to improve AI model training . Review event-driven architecture for correctness, including IAM/...SuggestedContract workSummer workLocal areaRemote workMercor
San Francisco, CA15 days ago$84 per hour
...General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position: Atomic Layer Deposition (ALD) Experts Type: Contract Compensation: $84/hour Location: Remote Commitment: 10–40 hours/week Role Responsibilities...Contract workSummer workRemote work$70 - $105 per hour
..., Larry Summers , and Jack Dorsey . Position: Biology Expert Type: Contract Compensation: $70–$105/hour Location... ...quality and downstream performance. Develop evaluation rubrics and benchmarks for assessing AI-generated biological tasks and...Contract workSummer workRemote workMercor Inc
San Francisco, CA2 days ago$100 per hour
...will work as a Healthcare Subject Matter Expert (SME) to create challenging, expert-... ...clinical content, and design high-quality rubrics defining what an accurate, complete, and... ...professional or clinical experience; senior and specialist-level professionals are strongly...Hourly payContract workFor contractorsRemote workSuperAnnotate
San Francisco, CA16 days ago$120 per hour
About the role We are hiring expert Evaluators in Customer success / support operations to review and assess AI-generated work... ...AI-generated artifacts against domain‑specific quality rubrics. Identify factual, aesthetic, and presentation errors. Provide...Hourly payWork at officeRemote workMercor
San Francisco, CA-54- ...Job Description Job Description Creative & Design Specialist Role Type: Contractor Location: Remote, Worldwide Schedule: Flexible... ..., annotation, RLHF, AI output evaluation, model evaluation, or rubric-based grading. ~ Strong understanding of professional...Remote jobFor contractorsWorldwideFlexible hours
YO AI Labs
San Francisco, CA8 days ago $80 - $120 per hour
...$80–$120/hour Location: Remote Role Responsibilities Evaluate AI-generated artifacts against domain-specific quality rubrics. Identify factual, aesthetic, and presentation errors in documents, spreadsheets, and slide decks. Provide clear, structured...Contract workSummer workWork at officeRemote workMercor
San Francisco, CA13 days ago$40 - $75 per hour
...Larry Summers , and Jack Dorsey . Position: Mathematics Expert Type: Contract Compensation: $40–$75/hour Location... ...quality and downstream performance. Develop evaluation rubrics and benchmarks for assessing AI-generated mathematical tasks...Contract workSummer workRemote workMercor
San Francisco, CA12 days ago- ...structured resolution of disagreements among chemistry reviewers. Develop and refine evaluation rubrics, review guidelines, and quality standards. Provide expert advisory support to client teams reviewing AI model performance. Write technically rigorous, publication...Contract workPart timeFor contractorsFreelanceRemote workFlexible hours
Turing
San Francisco, CA7 days ago $75 - $125 per hour
...Classify error types so the project team can see where models fail. Write grading rubrics and reference solutions at a consistent level of difficulty. Review problems written by other experts for correctness, clarity and originality. Provide clear written...Hourly payTemporary workFor contractorsRemote workRemoExperts
San Francisco, CA7 days ago- ...As a legal expert, you'll help train and evaluate how AI models reason about the law. This can be writing challenging legal prompts,... ...like fabricated citations or misapplied rules, drafting model answers and explanations, or creating rubrics to grade response quality....Temporary work
AfterQuery
San Francisco, CA7 days ago - ...As a biology expert, you'll help train and evaluate how AI models reason about biological sciences. This can be writing challenging... ...outdated nomenclature, or misapplied mechanisms, drafting model answers and explanations, or creating rubrics to grade response quality....Temporary work
AfterQuery
San Francisco, CA7 days ago $50 - $75 per hour
...own. Your standards and expertise become the rubric that frontier AI is graded against. This is a standing listing for specialists in consumer categories who want AI training... ...or several months. Finishing the domain expert interview and joining every network you...Odd jobPart timeLocal areaMercor
San Francisco, CA7 days ago- ...frontier AI labs use to test model judgment. AfterQuery builds the expert evaluation data that frontier AI labs use to measure and improve... ...Resume review. Every application gets read against a published rubric and gets a decision. Written assessment. You may be asked to...Weekly payContract workTemporary workFor contractorsRemote work10 hours per week
AfterQuery
San Francisco, CA7 days ago - ...Overview: We are seeking Materials Science experts to contribute advanced scientific... ...experienced researchers, faculty, and industry specialists interested in applying their expertise... ...Develop evaluation frameworks, scoring rubrics, and reference answers. Provide...Part time
Turing
San Francisco, CA2 days ago - ...test model judgment. AfterQuery builds the expert evaluation data that frontier AI labs... ...research, archival, or published work. Specialists in any period or region, with real primary... ...gets read against a published rubric and gets a decision. Written assessment....Weekly payContract workTemporary workFor contractorsRemote work10 hours per week
AfterQuery
San Francisco, CA7 days ago - ...semantics, and phonology. AfterQuery builds the expert evaluation data that frontier AI labs... ...of research, teaching, or applied work. Specialists in any subfield, syntax, semantics,... ...application gets read against a published rubric and gets a decision. Written assessment....Weekly payContract workTemporary workFor contractorsRemote work10 hours per week
AfterQuery
San Francisco, CA7 days ago - ...data. Role Overview We are seeking Board Game Reasoning Experts to help train and evaluate next-generation AI models. In this... ...correctness, consistency, and reasoning quality. Develop prompts, rubrics, and evaluation guidelines for board game and strategy-focused...Contract workPart timeFor contractorsFreelanceRemote work
Turing
San Francisco, CA7 days ago - ...frontier AI labs use to test model judgment. AfterQuery builds the expert evaluation data that frontier AI labs use to measure and improve... ...Resume review. Every application gets read against a published rubric and gets a decision. Written assessment. You may be asked to...Weekly payContract workTemporary workFor contractorsRemote work10 hours per week
AfterQuery
San Francisco, CA7 days ago - ...frontier AI labs use to test model judgment. AfterQuery builds the expert evaluation data that frontier AI labs use to measure and improve... ...Resume review. Every application gets read against a published rubric and gets a decision. Written assessment. You may be asked to...Weekly payContract workTemporary workFor contractorsRemote work10 hours per week
AfterQuery
San Francisco, CA7 days ago $25 per hour
...hourly, remote contractor role, you will work as a Persian Language Expert (SME) to review AI-generated Persian content and/or create high-... .../annotation, translation QA, editorial QA, localization QA, or rubric-based evaluation is strongly preferred. Skills: English,...Hourly payTemporary workFor contractorsRemote workShift workSuperAnnotate
San Francisco, CA7 days ago- ...Role Overview We are seeking Physics experts to contribute advanced scientific expertise... ...researchers, faculty, and industry specialists interested in applying their expertise to... ...Develop evaluation frameworks, scoring rubrics, and reference answers. Provide expert...Contract workPart timeFor contractorsFreelanceRemote workFlexible hours
Turing
San Francisco, CA2 days ago $50 per hour
...foundation model labs. Your expertise as an Chinese-Speaking Python Expert will directly help improve the world's premier AI models. Your... ...solutions, and rate multiple responses against consistent rubrics. This position is with SME Careers, a fast-growing AI data services...Hourly payFull timeContract workFor contractorsRemote workFlexible hoursSuperAnnotate
San Francisco, CA7 days ago$60 - $125 per hour
...generate on their own. Your standards and expertise become the rubric that frontier AI is graded against. This is a standing listing... ...can be within a week or several months. Finishing the domain expert interview and joining every network you qualify for both improve...Odd jobPart timeMercor
San Francisco, CA7 days ago- ...frontier AI labs use to test model judgment. AfterQuery builds the expert evaluation data that frontier AI labs use to measure and improve... ...Resume review. Every application gets read against a published rubric and gets a decision. Written assessment. You may be asked to...Weekly payContract workTemporary workFor contractorsRemote work10 hours per week
AfterQuery
San Francisco, CA7 days ago $120 per hour
...DAngelo Larry Summers and Jack Dorsey . Position: Legal Expert Corporate M&A Type: Contract Compensation: $... ...Adjudicate public commentary against task prompts and grading rubrics. Complete structural QC across prompts rubrics and...Full timeContract workSummer workRemote workMercor
San Francisco, CA9 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to JSON Specialist - Rubric Expert. Be the first to apply!
Related searches
- sewing specialist San Francisco, CA
- graduation specialist San Francisco, CA
- technology integration specialist San Francisco, CA
- vulnerability management specialist San Francisco, CA
- evaluation specialist San Francisco, CA
- escalation specialist San Francisco, CA
- educational technology specialist San Francisco, CA
- policy specialist San Francisco, CA
- transition specialist San Francisco, CA
- performance improvement specialist San Francisco, CA