AI Safety Specialist - Evaluation Expert

$60 - $70 per hour

Mercor

Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: AI Safety Practitioner
Type: Contract
Compensation: $60–$70/hour
Location: Remote
Role Responsibilities
Evaluate AI-generated responses for safety, factual accuracy, policy compliance, and overall quality.
Review content involving misinformation, political persuasion, self-harm, violence, cyber, biosecurity, and other sensitive domains.
Apply and refine evaluation rubrics for RLHF , SFT , and AI safety benchmarking .
Identify unsafe outputs, hallucinations, reasoning failures, and policy violations.
Provide structured feedback to improve model alignment and safety performance.
Collaborate with AI researchers and safety teams on ongoing evaluation initiatives.
Qualifications
Must-Have
Bachelor's degree or higher in Journalism, Communications, Psychology, Sociology, Public Policy, Law, Biology, Chemistry, Computer Science, or a related discipline.
5+ years of professional experience in AI Safety , Trust & Safety, journalism, public policy, scientific research, security, or a related field.
Excellent written English , critical thinking, and analytical reasoning skills.
Ability to consistently evaluate nuanced and policy-sensitive scenarios.
Preferred
Experience with AI Safety , RLHF , SFT , Trust & Safety, or AI evaluation.
Familiarity with safety policies, content moderation, or evaluation rubric development.
Experience reviewing complex, high-risk, or ambiguous content.
Application Process (Takes 20–30 mins to complete)
Upload resume
AI interview based on your resume
Submit form
Resources & Support
For details about the interview process and platform information, please check:
For any help or support, reach out to: View email address on us.fitly.work
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

Vacancy posted 12 days ago

Similar jobs that could be interesting for youBased on the AI Safety Specialist - Evaluation Expert in San Francisco, CA vacancy

Radiological Safety Expert - Redteaming - AI Trainer
$65 - $75 per unit
...technical talent with leading AI research labs. Headquartered in... .... Position: Radiologicals Safety for Redteaming Type:... ...dual-use, and adversarial. Evaluate model responses against a defined... ...responses. Collaborate with experts to ensure models correctly judge...
Suggested
Contract work
Summer work
Remote work
Mercor
San Francisco, CA
28 days ago
Frontend Code Evaluator - Expert - AI Trainer
$90 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...genuinely correct replications from those that merely look correct. Evaluate responsive behavior and semantic quality, ensuring proper use of...
Suggested
Contract work
Summer work
Local area
Remote work
Mercor
San Francisco, CA
28 days ago
Expert Interviewer - AI Trainer
$50 - $60 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Summers , and Jack Dorsey . Position: Expert Interviewer Type: Contract Compensation... ...technical depth, communication, and AI-evaluation fit using a provided rubric. Confirm...
Suggested
Contract work
Summer work
Remote work
Weekday work
Mercor
San Francisco, CA
18 days ago
Windows 11 Pro Expert - AI Trainer
$60 - $70 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Application Users - Windows 11 Professional Expert - Professionals Type: Contract Compensation... ...: Remote Role Responsibilities Evaluate and utilize Windows 11 Professional as...
Suggested
Contract work
Summer work
Remote work
Mercor
San Francisco, CA
12 days ago
Trainium Kernel Expert - AI Trainer
$70 - $90 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Dorsey . Position: Trainium (NKI) Kernel Expert Type: Contract Compensation: $... ...: Remote Role Responsibilities Evaluate the quality, correctness, and hardware-appropriateness...
Suggested
Contract work
Summer work
Remote work
Mercor
San Francisco, CA
12 days ago
Software Engineering Evaluator - Expert - AI Trainer
$100 - $150 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Dorsey . Position: Software Engineer Expert Type: Contract Compensation: $... ...Location: Remote Role Responsibilities Evaluate AI-generated artifacts for usability in...
Contract work
Summer work
Work at office
Remote work
Mercor
San Francisco, CA
20 days ago
Expert Senior Manager, AI Engineering
$293.5k
General Information Job Title Expert Senior Manager, AI Engineering Job ID 104335 Work... ...research, model experimentation, and evaluation design to production system... ...continual improvementBalance performance, safety, responsible AI principles, and cost...
Permanent employment
Full time
Apprenticeship
Work at office
Local area
Work from home
Home office
3 days per week
Bain & Company
San Francisco, CA
5 days ago
STEM Expert - Task Designer - AI Trainer
$2,000 per unit
...creative and technical talent with leading AI research labs. Headquartered in San... ..., and Jack Dorsey . Position: STEM Expert Task Pilot Type: Contract Compensation... ...AI capabilities. Ensure tasks are evaluated automatically from 0 to 1 with partial credit...
Contract work
Summer work
Remote work
Mercor
San Francisco, CA
5 days ago
Education Expert - AI Trainer
$60 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...-truth outputs and objective rubrics. Evaluate AI models on visual document understanding... ...content . Collaborate with subject matter experts to ensure task relevance and accuracy....
Contract work
Summer work
Remote work
Mercor
San Francisco, CA
12 days ago
Senior Safety Domain Expert - Falcon
...professional to serve as our Senior Safety Domain Expert for Veeva Falcon. In this... ...shaping next-generation AI and automation tools. You... ...Verification: Actively review, evaluate, and verify AI-driven... ...with PV leads and processing specialists at early adopter organizations...
Work at office
Local area
Work from home
Shift work
Veeva Systems
San Francisco, CA
2 days ago
Generalist Expert - Content Evaluator
$50 - $70 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Jack Dorsey . Position: Generalist Expert (US/Canada) Type: Contract Compensation... ...Role Responsibilities Review and evaluate documents, slides, spreadsheets, and similar...
Contract work
Summer work
Remote work
Flexible hours
Mercor
San Francisco, CA
27 days ago
Radiological Safety Expert - Red Team
$65 - $75 per hour
...technical talent with leading AI research labs.... ...Position: Emergency Response Specialist Type: Contract... ...in radiological safety, labeled as benign, dual... ...use, or adversarial. Evaluate AI model responses against... ...with radiological safety experts to assess misuse...
Contract work
Summer work
Remote work
Mercor
San Francisco, CA
12 days ago
Chemical Safety Expert - Redteaming
$65 - $75 per unit
...creative and technical talent with leading AI research labs. Headquartered in San... ...Jack Dorsey . Position: Chemicals Safety for Redteaming Type: Contract Compensation... ...as benign, dual-use, or adversarial. Evaluate model responses against defined policy...
Contract work
Summer work
Remote work
Mercor
San Francisco, CA
12 days ago
Remote Chemistry Expert — AI Training & Evaluation
$60 per hour
Prolific seeks Chemistry Experts and Chemical Engineers to join our Expert Network and evaluate AI models using your chemical expertise. Successful candidates will be invited to assess and review AI-generated chemistry tasks and ensure their accuracy. Compensation can reach...
Remote job
Hourly pay
Flexible hours
Prolific
San Francisco, CA
3 days ago
Finance & Strategy AI Engineer
$146.37k - $246k
.... At Samsara, we are helping improve the safety, efficiency and sustainability of the physical... ...of Finance’s data foundation and AI systems. Our team drives strategic decisions... ...orchestration frameworks, tool use, and evaluation techniques beyond prompt engineering Experience...
Full time
Remote work
Flexible hours
Samsara
San Francisco, CA
5 days ago
Frontend Code Evaluation Specialist - AI Trainer
$90 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...justification. Preferred Prior RLHF , preference-labeling, model-evaluation, or structured code-review work. Pixel-perfect design-to-code...
Contract work
Summer work
Local area
Remote work
Mercor
San Francisco, CA
12 days ago
AI Red Team Engineer
$182k - $238k
...agenda attempts to translate compute into safety at scale. Red-teaming previously sat... ...it now needs a dedicated owner. As the AI Red Team Engineer, you will help build the... ...scheming, detection of scheming (e.g. building evaluations), and scheming mitigations (e.g. anti-...
Full time
Work at office
Work from home
Visa sponsorship
Relocation package
Flexible hours
Apollo Research
San Francisco, CA
5 days ago
Senior AI Engineer
...Senior AI Engineer New York City or San Francisco | Hybrid... ...continuously improving clinical safety and accuracy. That means advancing... ..., retrieval algorithms, and evaluation infrastructure that allow... ...ground truth is incomplete, and experts disagree, and where the right...
Full time
Doctronic
San Francisco, CA
1 day ago
Music Producer - Audio Evaluation Expert
$39 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Jack Dorsey . Position: Music Audio Expert - Swedish Type: Contract Compensation... ...+ hours/week Role Responsibilities Evaluate AI model output lyrics , voice...
Contract work
Summer work
Immediate start
Remote work
Mercor
San Francisco, CA
12 days ago
AI Evaluation Engineer
$130k - $220k
...Artificial Analysis is the leading independent AI benchmarking and insights company. They... ...** This role is best described as an AI Evaluation Engineer / Technical Generalist. It is... ...success bar for this role is becoming a world expert in modern AI technologies. This is a...
Full time
Worldwide
Aurora Jobs ApS
San Francisco, CA
more than 2 months ago
Sales Engineering Expert - Evaluator
$100 - $150 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Dorsey . Position: Sales Engineering Expert Type: Contract Compensation: $... ...objections, supporting proofs of concept, and evaluating product fit and technical risk. ~ Hands...
Contract work
Summer work
Remote work
Mercor
San Francisco, CA
12 days ago
Software Engineer - Generative Data Platform, Evaluation
$150.4k - $225.3k
...Software Engineer - Generative Data Platform, Evaluation San Francisco, California, United... ...parts of the world where we operate.In the AI & Machine Learning (AIML) organization,... ...feature teams, ML data operations teams, and Safety and Responsible AI teams across Apple use...
Work experience placement
Relocation
Jobleads-US
San Francisco, CA
3 days ago
DevOps Engineer - AI Model Evaluator - AI Trainer
$400 per month
...About the Role Mercor is partnering with a leading AI research lab to support a Frontier Code Agents project. Contributors help evaluate and improve frontier AI coding models through structured technical assessments. The work focuses on realistic infrastructure engineering...
Mercor Inc
San Francisco, CA
1 day ago
AI Safety Expert - Red Team
$17 - $25 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey. Position: AI Safety Experts — English & Malay Type: Contract Compensation: $17–$25/hour...
Contract work
Summer work
Remote work
Mercor Inc
San Francisco, CA
2 days ago
Finance Expert - AI Evaluation Specialist
$60 - $100 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... .... Position: Finance Domain Expert — AI Training & Evaluation Type: Contract Compensation... ...with client researchers and specialists to maintain consistent standards. Translate...
Hourly pay
Contract work
For contractors
Summer work
Mercor
San Francisco, CA
12 days ago
Senior/Staff Applied AI Engineer
...Echelon Echelon is building the AI platform for Business... ...agent loops, tool-use patterns, evaluation systems, and multi-agent coordination... ..., factuality, tool selection, safety, latency, and cost. Improve... ...Engineering and domain experts to turn real BizOps workflows...
Work at office
Jobleads-US
San Francisco, CA
3 days ago
Staff AI Engineer: Build & Evaluate Autonomous Agents
...Raydar is seeking a Staff AI Engineer in San Francisco to design and ship software systems layered on top of large language models. You will own major parts of the product, focusing on evaluating AI agents, agentic concepts, and production stability. Requires hands-on...
Full time
Jobleads-US
San Francisco, CA
2 days ago
AI Safety Specialist - Fully Remote | Upto $62/hr
$62 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Angelo , Larry Summers , and Jack Dorsey . Position: AI Safety Experts — English & Dutch Type: Contract Compensation: $48–$...
Contract work
Summer work
Remote work
Mercor
San Francisco, CA
6 days ago
AI Platform Engineer
...and operational knowledge into governed AI workflows that act in production. Our... ...machines, fleets, suppliers, field teams, safety constraints, and decisions with consequences... ...model routing, tool-use, memory, and evaluation capabilities. Work directly with customers...
Redapto
San Francisco, CA
4 days ago
Founding AI Research Engineer - Robot Learning
...Origin is building Physical AI for the built world - starting... ...commercial projects while improving safety and predictability on the job... ...runs a Multi Agent Action Expert architecture: classical precision... ...detection. Design offline evaluation metrics that predict real-...
Origin
San Francisco, CA
a month ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Safety Specialist - Evaluation Expert. Be the first to apply!