Bilingual Language Model Evaluator
$15 - $20 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Generalist - English & Urdu
Type: Contract
Compensation: $15–$20/hour
Location: Remote
Role Responsibilities
- Conduct fact-checking using trusted public sources and external tools .
- Generate high-quality human evaluation data by identifying response strengths, areas for improvement, and factual inaccuracies.
- Assess reasoning quality, clarity, tone, and completeness of responses.
- Ensure model responses align with expected conversational behavior and system guidelines.
- Work independently and asynchronously to meet deadlines while improving AI model performance .
Qualifications
Must-Have
- Bachelor's degree .
- Native speaker in Urdu .
- Significant experience using large language models (LLMs).
- Excellent writing skills in English .
- Strong attention to detail .
- Background or experience in domains requiring structured analytical thinking .
Preferred
- Prior experience with RLHF, model evaluation, or data annotation work .
- Experience writing or editing high-quality written content .
- Experience comparing multiple outputs and making fine-grained qualitative judgments .
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on ziprecruiter.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Bilingual Language Model Evaluator in San Francisco, CA vacancy
- .... You will be responsible for fact-checking, generating evaluation data, and assessing model responses for clarity and completeness. The ideal candidate... ...'s degree, has significant experience with large language models, and possesses excellent English writing skills....BilingualRemote job
- ...Welo Data is seeking Data Labeling Associates in California to evaluate AI outputs and ensure cultural context and safety in Arabic datasets. This role requires professional-level proficiency in Portuguese (Brazil), a bachelor's degree, and at least 2 years of experience...Bilingual
Welo Data
San Francisco, CA4 days ago $28 - $60 per hour
...Allcaretherapies is hiring part-time Speech Language Pathologists to provide remote therapy services across California, Nevada, and... ...license. Compensation ranges from $28 to $60 per hour, depending on bilingual proficiency and experience. Join a collaborative team that...BilingualHourly payPart timeRemote workAllcaretherapies
San Francisco, CA2 days ago- ...Francisco, California. The Role: As a Research Engineer - Language Model Pre-Training , you'll shape our language model roadmap... ...our pretraining stack Dataset collection, processing, and evaluation Architecture and methodology research, including optimizer...SuggestedWork at officeRelocation package
$30 - $46 per hour
...Speech Language Pathology Assistant (SLPA) - Remote All Care Therapies is a leading provider of Physical, Occupational, Speech, and... ..., behavioral modifications, adults, and AAC (preferred) Bilingual and able to treat in Spanish (preferred) Hours and Location...BilingualHourly payPart timePrivate practiceLocal areaRemote workWork from homeFlexible hoursAll Care Therapies
San Francisco, CA5 days ago$89k - $105.7k
...Public Schools. We offer a Speech Language Pathology team lead that can... ...for special education. Bilingual (Spanish or Vietnamese) preferred... ...to perform diagnostic evaluations according to CA law and ASHA... ...literacy achievement and to model scaffolding strategies. Support...BilingualFull timeTemporary workWork at officeKIPP Public Schools Northern California
San Francisco, CA2 days ago$301.75k - $355k
...This Role The Senior Director for the Model LifeCycle team will undertake a pivotal... ...Machine Learning models, including Large Language Models (LLMs). What You’ll Be Working... ...management: versioning, lineage, evaluation, and reproducible fine‑tuning at scale....Temporary workEpoch Biodesign
San Francisco, CA2 days ago$43 - $56 per hour
...California licensure! We are actively seeking part-time Speech Language Pathologists (SLPs) to join our Provider Network. You will... ...00 per hour commensurate with experience, qualifications, and bilingualism. *Reimbursement for licensure(s) will be paid out after 145...BilingualHourly payPart timeLocal areaRemote workWork from homeFlexible hoursAll Care Therapies
San Francisco, CA5 days ago$58 - $63 per hour
...Research Intern, Model Shaping (Fall 2026) San Francisco, Amsterdam About The Role... ...methods) Robust and reliable evaluation of foundation model capabilities The... ...ideas across machine learning, natural language processing, and ML systems. Past research...Hourly payInternshipWork at officeTogether AI
San Francisco, CA2 days ago- ...Crusoe in San Francisco is seeking a Senior Director for the Model LifeCycle team to establish a managed platform for the development... ..., and expertise in managing fine-tuning systems for large language models. A competitive compensation package and comprehensive benefits...Full time
Crusoe
San Francisco, CA1 day ago $70.3k - $76.76k
...serve. The Position The Speech and Language Pathology Assistant will work under the... ...elementary school students in screening, evaluating, diagnosing, and treating disorders of... ...the State of California (required). Bilingual in English and Spanish or Vietnamese (strongly...BilingualLocal areaRocketship Public Schools
San Francisco, CA1 day ago- ...Member of Technical Staff, Model Efficiency Who are we? Our mission is to scale intelligence to serve humanity. We’re training... ...Python (Rust/Go also welcome) ~ Experience working with large language models and familiarity with the LLM inference ecosystem (e.g.,...Full timeWork at officeRemote workFlexible hours
Cohere
San Francisco, CA1 day ago $100k - $115k
...Speech and Language Pathologist Felton ECE Programs Agency Overview Felton Institute... ...Responsibilities Screen, evaluate and treat children with communication needs... ...assessing potential language delays Bilingual in Spanish/English required...BilingualFull timeContract workLocal areaFelton Institute
San Francisco, CA1 day ago$85 per hour
...Location: Remote Role Responsibilities Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks. Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , and infrastructure...Contract workSummer workRemote work$85 per hour
...$85/hour Location: Remote Role Responsibilities Use frontier AI coding agents to complete and evaluate complex engineering tasks. Review model-generated mobile application code for correctness, quality, maintainability, and performance. Identify bugs...Contract workSummer workRemote work$298k - $368k
...continuously learning from large scale real-world data, to (2) develop models and model training at scale, to (3) analyze real-world behavior... ...and technical constraints ~ Experience applying large language models or foundation models in complex, safety-critical domains...Full timeRemote workWaymo
San Francisco, CA1 day ago$144k - $187k
...and frameworks that power MSCI's quantitative risk and factor model research. The team develops the systems and environments that enable... ...participants across the financial ecosystem with a common language. Our research-based data, analytics and indexes, supported by advanced...Flexible hoursMSCI
San Francisco, CA1 day ago$220k - $320k
...About Inference.net Inference.net trains and hosts specialized language models for companies that need frontier‑quality AI at a fraction of... ...handles everything end‑to‑end: distillation, training, evaluation, and planet‑scale hosting. We are a well‑funded ten‑person...Work at office- ...seeking a compassionate and skilled Speech-Language Pathologist (SLP) to join our... ...Documentation Conduct initial and ongoing evaluations, screenings, and assessments to identify... ..., and communication skills Bilingual proficiency (strongly preferred) Salary...BilingualRelocation
- ...seeking a compassionate and skilled Speech-Language Pathologist (SLP) to join our... ...Documentation Conduct initial and ongoing evaluations, screenings, and assessments to identify... ..., and communication skills Bilingual proficiency (strongly preferred) Salary...BilingualRelocation
- A leading AI research firm in San Francisco is seeking a Member of Technical Staff specialized in Model Efficiency. In this role, you will enhance LLM inference systems by tackling performance issues and collaborating with cross-functional teams. Ideal candidates have over...Full timeRemote work
Cohere
San Francisco, CA1 day ago $120k - $180k
A cutting-edge tech startup is seeking a Founding Software Engineer to build systems that leverage physics-informed models. The ideal candidate will have strong software engineering skills, experience in ML/AI application development, and a mindset geared towards ownership...Full timeGodela (YC X25)
San Francisco, CA1 day ago$180k - $260k
...A leading AI research company is seeking a Model Behavior Architect to design and refine strategies for model behavior across various... ...design, and engineering teams, focusing on crafting effective evaluation pipelines and ensuring alignment with product goals. Ideal candidates...Full timePerplexity
San Francisco, CA1 day ago- Welo Data is seeking Data Labeling Associates in San Francisco to evaluate AI systems focused on Arabic language nuances. The role includes model evaluation, identifying biases in datasets, and ensuring quality across global AI workflows. Candidates should have native French...
$170k - $216k
...engineers like you to (1) develop methods for efficiently and continuously learning from large scale real-world data, to (2) develop models and model training at scale, to (3) analyze real-world behavior and develop systems for handling the complexities of interacting...Full timeRemote workWaymo
San Francisco, CA1 day ago$115.7k - $119.5k
...consulting along with technology and design, corporate and digital ventures—and business purpose. We work in a uniquely collaborative model across the firm and throughout all levels of the client organization, generating results that allow our clients to thrive. As a...Work at officeLocal areaBoston Consulting Group
San Francisco, CA3 days ago- ...Francisco seeks a Machine Learning Engineer to join their AI/ML team. The role involves creating innovative AI experiences, training models, and developing unique features to wow customers. Candidates should have extensive experience in NLP and ML Ops with a strong...Full time
Lightfield
San Francisco, CA1 day ago $80 - $120 per hour
Mercor, based in San Francisco, is seeking a Healthcare operations Evaluator to evaluate AI-generated artifacts against quality standards and provide feedback. Candidates must have more than 5 years of experience in Healthcare operations and fluency in English. This contract...Remote jobContract workWork at office$19.61 - $25.5 per hour
...retail showroom presents a professional and inviting appearance. Bilingual candidates encouraged to apply. ESSENTIAL JOB FUNCTIONS... ...Ability to drive a manual transmission vehicle Fluency in multiple languages (Spanish is highly desired) O’Reilly Auto Parts has a proven...BilingualHourly payMinimum wagePart timeLocal areaFlexible hours$80 - $120 per hour
Mercor is seeking a Software / AI / IT / data Evaluator to join their team remotely. In this role, you will evaluate AI-generated artifacts based on quality standards, providing structured feedback and ensuring accuracy. The ideal candidate will have over 5 years of experience...Remote jobHourly payContract workWork at office
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Bilingual Language Model Evaluator. Be the first to apply!
Related searches
- evaluator San Francisco, CA
- education evaluator San Francisco, CA
- quality evaluator San Francisco, CA
- ai evaluator San Francisco, CA
- social media evaluator San Francisco, CA
- bilingual San Francisco, CA
- bilingual school San Francisco, CA
- japanese bilingual San Francisco, CA
- bilingual director San Francisco, CA
- bilingual spanish customer service San Francisco, CA