Applied ML Task Auditor & Rubric Evaluator
Obsidian
Obsidian is seeking an evaluation-focused ML scientist to assess applied ML rigor in frontier AI research settings. You will critique experiment designs, model-selection reasoning, and evaluation methodologies, delivering rubric-based feedback to improve reproducibility and validity. Responsibilities emphasize identifying data leakage, metric gaming, and CV hygiene issues, ensuring claims are well-supported by evidence and experiments are replicable across datasets and baselines. #J-18808-Ljbffr Obsidian
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Applied ML Task Auditor & Rubric Evaluator in San Francisco, CA vacancy
- Evaluate the quality, correctness, and methodological rigor of applied machine-learning tasks used to train and evaluate a frontier AI lab's models.... ...methodology — and provide clear, rubric-based written feedback.... ...‑on applied/experimental ML (experiment design, model selection...Suggested
- Mercor is seeking a reviewer to evaluate the quality, correctness, and methodological rigor of applied machine-learning tasks used to train and evaluate... ...methodology, and deliver rubric-based written feedback. The... ...practices, critique of ML claims against evidence, and...Suggested
$117k - $160.6k
...talk.As a Senior IT Internal Auditor, you will serve as a technical... ...who can move beyond task execution: you will contextualize... ...to the technology environment.Apply professional judgment in setting... ...efficiency, automate workpapers, and evaluate AI/ML controls and...SuggestedWork at officeLocal areaWorldwideFlexible hoursShift work2 days per week- Become a Luxury Brand Evaluator in San Francisco - Apply Now Turn your passion for luxury into a career opportunity. Explore the world of premium brands and make a lasting impact in fashion, beauty, jewelry, or automobiles. Join CXG, the global leader in customer experience...SuggestedContract workWorldwideFlexible hours
- Current Employees of Happen Bank: Please apply via your internal Workday AccountHappen Bank... ..., and remediation trackingTest and evaluate key ITGCs such as user access reviews and... ...such as AWS, Azure, or GCPExposure to AI/ML governance or model risk conceptsExperience...SuggestedFull timeWork at officeLocal areaRemote workRelocationFlexible hours
- Mercor seeks expert Evaluators in User/customer research and feedback synthesis to review AI... ..., rigor, and domain quality. You will apply deep subject-matter expertise to grade outputs... ...will evaluate artifacts against quality rubrics and identify factual, aesthetic, and...Remote jobHourly pay
- Obsidian is seeking expert Evaluators in Operations / inventory / capacity planning to review... ...hourly, inviting professionals who can apply deep subject-matter expertise to grade... ...will evaluate artifacts against quality rubrics, identify factual and presentation errors...Remote jobHourly pay
- Obsidian is seeking expert Evaluators in Humanities / arts / culture to review AI-generated... ...products for accuracy and quality. You will apply deep subject-matter expertise to grade... ...meticulous attention to domain quality rubrics. A Master's degree is preferred and...Remote jobHourly pay
$120 per hour
About the role We are hiring expert Evaluators in Real estate / hospitality / events to review... ..., rigor, and domain quality. You will apply deep subject-matter expertise to grade... ...artifacts against domain-specific quality rubrics. Identify factual, aesthetic, and...Remote jobHourly payWork at office- Obsidian evaluates Neuron Kernel Interface (NKI) development tasks used for frontier AI model training and evaluation. The role focuses on assessing CUDA→NKI migration... ...-correctness standards. The candidate will deliver rubric-based feedback with concrete recommendations to...
- ...seeking experienced AI Safety Practitioners to evaluate the safety, quality, and alignment of... ...You will assess AI-generated responses, apply safety policies, and help improve model... ...sensitive domains. Apply and refine evaluation rubrics for RLHF, SFT, and AI safety benchmarking...Worldwide
- ...experienced Tax Professionals and US Tax Law Experts to apply expertise in training and evaluating AI systems on US sales and use tax statutes. You will... ...rules. Responsibilities include creating evaluation rubrics, assigning severity levels, and reviewing AI-generated...Remote job
- ...sales engineers for a project focused on evaluating how well AI systems perform real-world technical... ...excellent work looks like: designing task-specific grading criteria and scoring... ...detailed written justifications for every score Apply consistent, evidence‑based judgment so...
- Evaluate the quality, correctness, and production-readiness of Kubernetes tasks used to train and evaluate a frontier AI lab's models. You'll assess cluster-operations scenarios... ...-mode troubleshooting — and provide clear, rubric-based written feedback. Basic Qualifications...
- ...estimated costs for large repairs and improvements. Act as an authority for specialties needed. Apply general knowledge of standard principles and techniques/procedures to accomplish tasks and solve routine problems. Possess a wide understanding of own job field and some...
$400 per month
...Frontier Code Agents project. Contributors help evaluate and improve frontier AI coding models... ...complex infrastructure engineering tasks. Review model-generated implementations involving... ...assess their strengths and weaknesses. Apply professional engineering judgment to...Mercor Inc
San Francisco, CA12 hours ago- Current Employees of Happen Bank: Please apply via your internal Workday AccountHappen Bank... ....About the RoleWe are looking for a Sr Auditor who will thrive in a fast-paced... ...reporting and validationIdentify key risks and evaluate control design and effectiveness in a regulated...Full timeWork at officeLocal areaRemote workRelocationFlexible hours
- Mercor is seeking expert Evaluators in Legal/compliance to review AI-generated documents, spreadsheets, and slide decks for accuracy and... ...and careful judgment. You will assess artifacts against rubrics, identify errors, and provide structured written feedback to improve...Remote jobHourly payWork at office
- Obsidian seeks experienced AI Safety Practitioners to evaluate safety, quality, and alignment of frontier AI models across grey-area topics. You will assess AI-generated responses, apply safety policies, and help improve model behavior through structured evaluations and...
- Obsidian is seeking expert Evaluators in Compliance/regulatory response with financial-services AI to review AI-generated documents, spreadsheets... ...slide decks for accuracy, rigor, and domain quality. You will apply deep subject-matter expertise to grade outputs. This remote,...Remote jobHourly payFlexible hours
- Obsidian is seeking expert Evaluators in Legal/compliance to review AI-generated documents, spreadsheets, and slide decks for accuracy, rigor, and domain quality. You will apply deep subject-matter expertise to grade outputs. This is a remote, hourly engagement. You will...Remote jobHourly pay
$101k - $155k
...Tanium's audit program, leading a team of auditors, developing audit methodology, and... ...timeframe and budgetConfident delegating tasks, and consistent in tracking and monitoring... ...DORA, ENS, EU AI Act and GDPRSkilled in applying technical and program-management expertise...Full timeLive inWorldwideFlexible hours- ...Tuesday. We are seeking an experienced Senior Auditor - Finance and Operations to join our... ...timelines. Identify control deficiencies, evaluate severity and root cause, and partner with... ...and validate effective remediation. Apply professional skepticism to independently...Contract workWork at officeLocal areaWork from homeFlexible hours
Lambda
San Francisco, CA5 days ago $124k - $280k
.... You will motivate and coach teams to solve complex problems, applying sound judgment to recognize when to take action and when to escalate... ...our firm.Responsibilities- Leading internal audit projects to evaluate and enhance financial controls and compliance- Developing and...Full timeH1b$119k - $299.93k
...coach teams to solve complex problems, applying sound judgment to know when to take action... ...digital assurance projects focused on evaluating clients' digital environments, including... ...license or Certified Information Systems Auditor (CISA) certificationWhat Sets You Apart-...Full timeH1b- ...Frontier Code Agents project. You will help evaluate frontier AI coding models by performing infrastructure engineering tasks and reviewing model-generated implementations... ...with professional engineering judgment, applying real-world reliability concepts to production...
- Mercor is recruiting experienced musicians to evaluate generative music AI models in partnership with a leading AI lab. You will assess AI... .... Start date is immediate; position is hourly with potential transition to task-based pay within up to 6 #J-18808-Ljbffr MercorHourly payImmediate start
$78.68k - $157.88k
Position Summary Audit and Assurance Information Technology Auditor - Senior Consultant Do you thrive in times of disruption? Have a... ..., cyber, and regulatory risks—to gain competitive advantage. We apply our experience in ongoing business operations and corporate...Work experience placementWork at officeLocal areaVisa sponsorship- Mercor is partnering with a leading AI research organization to evaluate how well AI systems perform real-world marketing work. You will define what excellent work looks like: designing task-specific grading criteria and scoring completed work samples with rigorous, well...
- Overnight Hotel Self Parking Lot Auditor - San Francisco, CA San Francisco, CA, USA Job Description Posted Tuesday, July 28, 2026 at 4:... ...Responsibilities Support evolving business needs by completing other tasks as assigned, attending required meetings and training, and...Hourly payMinimum wageFull timePart timeLocal areaShift workNight shiftWeekend work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Applied ML Task Auditor & Rubric Evaluator. Be the first to apply!
Related searches
- information system auditor San Francisco, CA
- coding auditor San Francisco, CA
- security auditor San Francisco, CA
- energy auditor San Francisco, CA
- medical records auditor San Francisco, CA
- inventory auditor San Francisco, CA
- remote auditor San Francisco, CA
- data auditor San Francisco, CA
- sales auditor San Francisco, CA
- auditor San Francisco, CA