AI Safety Practitioner - Expert Evaluator
$60 - $70 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: AI Safety Practitioner
Type: Contract
Compensation: $60–$70/hour
Location: Remote
Role Responsibilities
- Evaluate AI-generated responses for safety, factual accuracy, policy compliance, and overall quality.
- Review content involving misinformation, political persuasion, self-harm, violence, cyber, biosecurity, and other sensitive domains.
- Apply and refine evaluation rubrics for RLHF , SFT , and AI safety benchmarking .
- Identify unsafe outputs, hallucinations, reasoning failures, and policy violations.
- Provide structured feedback to improve model alignment and safety performance.
- Collaborate with AI researchers and safety teams on ongoing evaluation initiatives.
Qualifications
Must-Have
- Bachelor's degree or higher in Journalism, Communications, Psychology, Sociology, Public Policy, Law, Biology, Chemistry, Computer Science, or a related discipline.
- 5+ years of professional experience in AI Safety , Trust & Safety, journalism, public policy, scientific research, security, or a related field.
- Excellent written English , critical thinking, and analytical reasoning skills.
- Ability to consistently evaluate nuanced and policy-sensitive scenarios.
Preferred
- Experience with AI Safety , RLHF , SFT , Trust & Safety, or AI evaluation.
- Familiarity with safety policies, content moderation, or evaluation rubric development.
- Experience reviewing complex, high-risk, or ambiguous content.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on us.fitly.work
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 10 days ago
Similar jobs that could be interesting for youBased on the AI Safety Practitioner - Expert Evaluator in San Francisco, CA vacancy
- Obsidian is seeking expert Evaluators in FP&A / corporate finance to assess AI-generated work products for accuracy and quality. This role entails deep expertise to grade outputs and provide structured feedback. Candidates should have at least 5 years of relevant experience...SuggestedRemote jobHourly payWork at office
- Obsidian is seeking expert Evaluators in Biology/environmental science to review and assess AI-generated work products for accuracy and quality. In this remote, hourly role, you will leverage your expertise to provide feedback on documents and presentations, ensuring they...SuggestedRemote jobHourly pay
- ...Welo Data is seeking Data Labeling Associates in California to evaluate AI outputs and ensure cultural context and safety in Arabic datasets. This role requires professional-level proficiency in Portuguese (Brazil), a bachelor's degree, and at least 2 years of experience...Suggested
Welo Data
San Francisco, CA2 days ago - Obsidian seeks experienced AI Safety Practitioners to evaluate safety, quality, and alignment of frontier AI models across grey-area topics. You will assess AI-generated responses, apply safety policies, and help improve model behavior through structured evaluations and...Suggested
- Mercor is seeking experienced AI Safety Practitioners to evaluate frontier AI models for safety, alignment, and policy compliance across nuanced topics. You will assess AI-generated responses for quality, identify unsafe outputs, and apply robust evaluation rubrics for...Suggested
- ...Francisco is hiring Data Labeling Associates for Project Perseus. This role focuses on evaluating Arabic AI systems, requiring professional proficiency in Portuguese and experience in AI safety. Responsibilities include assessing AI outputs, identifying bias, and...Full time
$70 - $84 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ..., and Jack Dorsey . Position: AI Safety Red Teamer Type: Contract Compensation... ...hallucinations, and policy failures. Evaluate model robustness across misinformation, cyber...Contract workSummer workRemote workMercor
San Francisco, CA9 days ago- We are seeking experienced AI Safety Red Teamers to identify vulnerabilities in frontier AI systems through adversarial testing. You will design challenging prompts, uncover model weaknesses, and evaluate AI behavior across complex, high-risk, and ambiguous ("grey-area"...
- ...Mercor, we believe the safest AI is the one that’s already been... ...for this project - human data experts who probe AI models with adversarial... ...customer AI systems Evaluation coverage expands: more scenarios... ...production Mercor customers trust the safety of their AI because you’ve...Remote work
- We are seeking experienced AI Safety Practitioners to evaluate the safety, quality, and alignment of frontier AI models across complex, policy-sensitive, and ambiguous ("grey-area") topics. You will assess AI-generated responses, apply safety policies, and help improve...Worldwide
- ...Data is seeking Data Labeling Associates in San Francisco to evaluate AI systems focused on Arabic language nuances. The role includes... ...like gourmet dining and comprehensive medical coverage while contributing to innovative AI safety solutions. #J-18808-Ljbffr Welo Data
$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... .... Position: User/customer research and feedback synthesis Evaluator Type: Contract Compensation: $80–$120/hour Location...Contract workSummer workWork at officeRemote work$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Position: Government / public administration Evaluator Type: Contract Compensation: $... .... Collaborate with subject matter experts to ensure consistency and quality. Work...Contract workSummer workWork at officeRemote work$90 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...genuinely correct replications from those that merely look correct. Evaluate responsive behavior and semantic quality, ensuring proper use of...Contract workSummer workLocal areaRemote work- ...intraoperative needs under physician orders. You will document perioperative care, coordinate with surgeons and anesthesiologists, and perform certain procedures outside the OR according to KP policy, maintaining patient safety and compliance. #J-18808-Ljbffr Kaiser Permanente
$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Position: Humanities / arts / culture Evaluator Type: Contract Compensation: $... ...performance . Collaborate with subject matter experts to ensure consistency and quality....Contract workSummer workWork at officeRemote workMercor
San Francisco, CA4 days ago$50 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Jack Dorsey . Position: Spanish (Spain) Audio Generalist Evaluator Expert Type: Contract Compensation: $50/hour Location:...Contract workSummer workRemote workMercor
San Francisco, CA9 days ago- About the roleWe're building a high-quality evaluation dataset for CNC manufacturing and are looking for experienced CNC machinists to help author and validate grading rubrics for CNC machining work. You'll bring real production-floor judgment to determine whether a machining...
- Mercor is hiring experienced musicians to evaluate generative music AI models, in partnership with a leading AI lab. You will assess AI-generated music across genres and rate it against detailed quality standards, working in Hindi and English. Bring a background in songwriting...
$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...and Jack Dorsey . Position: BI dashboards / performance reporting Evaluator Type: Contract Compensation: $80-$120/hour Location: Remote Role...Contract workSummer workWork at officeRemote work$48 - $62 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Angelo , Larry Summers , and Jack Dorsey . Position: AI Safety Experts — English & Danish Type: Contract Compensation: $48–...Contract workSummer workRemote work$80 - $150 per hour
...technical talent with leading AI research labs. Headquartered in... ...Health Clinical Advisor (AI Safety Benchmark Project) Type:... ...and refine clinician-informed evaluation rubrics for AI chatbot responses... ...generated conversations and provide expert judgment to calibrate...Contract workTraineeshipSummer workRemote work$24 - $35 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Angelo , Larry Summers , and Jack Dorsey . Position: AI Safety Experts — English & Thai Type: Contract Compensation: $24–$3...Contract workSummer workRemote work- 1. Role Overview Mercor is partnering with a leading AI research organization to engage experienced sales engineers for a project focused on evaluating how well AI systems perform real-world technical sales work. Rather than producing deliverables yourself, you will define...
- Obsidian is looking for expert Evaluators in Finance operations/audit support to review AI-generated work products for accuracy and quality. This remote hourly position requires a minimum of 5 years in finance and fluency in English. Your role will involve evaluating outputs...Remote jobHourly payWork at office
$80 - $120 per hour
Mercor is seeking a Media / journalism / communications Evaluator to evaluate AI-generated artifacts and provide structured feedback. Ideal candidates will have over 5 years of experience and must be fluent in English. This remote role requires proficiency in Microsoft...Remote jobHourly payWork at office- Obsidian is hiring expert Evaluators in Real estate, hospitality, and events to review AI-generated work for accuracy, rigor, and domain quality. This remote position requires deep expertise and involves grading outputs like documents and presentations. Applicants must...Remote jobWork at office
- Obsidian is seeking expert Evaluators in Legal/compliance to review AI-generated documents, spreadsheets, and slide decks for accuracy, rigor, and domain quality. You will apply deep subject-matter expertise to grade outputs. This is a remote, hourly engagement. You will...Remote jobHourly pay
$80 - $120 per hour
Mercor is seeking a User/Customer Research and Feedback Synthesis Evaluator to evaluate AI-generated artifacts using specific quality rubrics. The role demands deep subject-matter expertise in user research and involves providing feedback to improve AI performance. The...Remote jobContract workWork at officeFlexible hours- Mercor is seeking experienced musicians to evaluate generative music AI models in partnership with a leading AI lab. You will assess AI-generated lyrics across a wide range of genres and rate them against detailed quality standards, working in Malayalam and English. Required...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Safety Practitioner - Expert Evaluator. Be the first to apply!
Related searches
- practitioner San Francisco, CA
- infection control practitioner San Francisco, CA
- holistic practitioner San Francisco, CA
- subject matter expert San Francisco, CA
- fulfillment expert San Francisco, CA
- guest service support expert San Francisco, CA
- technology expert San Francisco, CA
- ai evaluator San Francisco, CA
- work from home web search evaluator San Francisco, CA
- social media evaluator San Francisco, CA