Bilingual Language Model Evaluator
$15 - $20 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Generalist - English & Urdu
Type: Contract
Compensation: $15–$20/hour
Location: Remote
Role Responsibilities
- Conduct fact-checking using trusted public sources and external tools .
- Generate high-quality human evaluation data by identifying response strengths, areas for improvement, and factual inaccuracies.
- Assess reasoning quality, clarity, tone, and completeness of responses.
- Ensure model responses align with expected conversational behavior and system guidelines.
- Work independently and asynchronously to meet deadlines while improving AI model performance .
Qualifications
Must-Have
- Bachelor's degree .
- Native speaker in Urdu .
- Significant experience using large language models (LLMs).
- Excellent writing skills in English .
- Strong attention to detail .
- Background or experience in domains requiring structured analytical thinking .
Preferred
- Prior experience with RLHF, model evaluation, or data annotation work .
- Experience writing or editing high-quality written content .
- Experience comparing multiple outputs and making fine-grained qualitative judgments .
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on ziprecruiter.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 9 days ago
Similar jobs that could be interesting for youBased on the Bilingual Language Model Evaluator in San Francisco, CA vacancy
- ...remotely. The role focuses on fact-checking and generating evaluation data, requiring native fluency in Urdu and strong English writing... ...a bachelor’s degree and significant experience with large language models. Responsibilities also include independently assessing...BilingualRemote job
- .... You will be responsible for fact-checking, generating evaluation data, and assessing model responses for clarity and completeness. The ideal candidate... ...'s degree, has significant experience with large language models, and possesses excellent English writing skills....BilingualRemote job
- ...detail-oriented Data Quality Associate in San Francisco. This role involves evaluating AI model outputs, providing feedback, and ensuring data quality. Candidates must have native-level language proficiency and a Bachelor's degree. The position requires strong critical...Suggested
- The Opportunity As an Applied Scientist specializing in Small Language Models and AI Training, you will lead research and development... ...small and efficient language models. Design, implement, and evaluate model training experiments to improve performance, robustness...Suggested
$50 - $75 per hour
A leading tech company based in Australia is seeking an AI Model Evaluator on a contract basis. The role involves evaluating AI-generated responses, writing prompts, and providing justifications based on specific criteria. Ideal candidates will hold a Master's degree in...SuggestedHourly payContract work- Obsidian is seeking an independent contractor for a remote role focused on benchmark dataset projects that evaluate AI models on visual document understanding in the Energy/Utilities domain. The role requires authoring complex tasks and providing objective rubrics to assess...Remote jobFor contractorsFlexible hours
- ...act on challenges in real time. What You’ll Do As a Vision Language Model Engineer, you will design, develop, and optimize advanced vision... ...in real-world applications. Conduct experiments to evaluate model performance and iterate on architectures and training pipelines...Flexible hours
- Welo Data is seeking Data Labeling Associates in California to evaluate AI outputs and ensure cultural context and safety in Arabic datasets. This role requires professional-level proficiency in Portuguese (Brazil), a bachelor's degree, and at least 2 years of experience...Bilingual
- A cutting-edge AI company in San Francisco is looking for a technical owner for enterprise vision-language models (VLMs). This role includes end-to-end management of VLM post-training projects and collaborating directly with customers. Ideal candidates have hands-on experience...
- A cutting-edge AI firm in San Francisco seeks a VLM Post-Training Owner to lead enterprise engagements and enhance vision-language models. The role combines project ownership with technical execution, ensuring quality data generation and customer satisfaction in AI solutions...
$62.19k - $101.85k
...Common Core Standards across multiple subjects and teach in a Dual-Language Spanish Immersion Program. Qualifications include a Bachelor's... ...accredited university, a California teaching credential, and bilingual abilities in Spanish and English. This position offers a...Bilingual$20 - $26 per hour
Prolific is seeking fluent Thai speakers to join their project as evaluators. You will assess AI-generated audio for naturalness and... ...and remote work. This is a unique opportunity to influence future AI models with your linguistic expertise. #J-18808-Ljbffr ProlificRemote jobHourly payFlexible hours$75 - $85 per hour
...services utilizing the Parent Coaching Model (Rush & Sheldon) Follow each child’s Individualized... ...) Requirements: California Speech Language Pathology license Pediatric experience... ...Strong clinical & interpersonal skills Bilingual in Spanish a plus! Sunny Days is an...BilingualPart timeLocal areaWork from homeRelocation packageFlexible hoursSunny Days Early Childhood Developmental Services
San Francisco, CA1 day ago- ...Bienestar (Well-Being), Culture of Belonging, Community Partnership, Agency & Self-Determination, and Joy. Job Summary The Bilingual Speech-Language Pathologist provides a full range of speech and language services to children in grades K-12, including screening and...BilingualWork at office
Aspire Public Schools
San Francisco, CA1 day ago $43 - $56 per hour
...California licensure! We are actively seeking part-time Speech Language Pathologists (SLPs) to join our Provider Network. You will... ...00 per hour commensurate with experience, qualifications, and bilingualism. *Reimbursement for licensure(s) will be paid out after 145...BilingualHourly payPart timeLocal areaRemote workWork from homeFlexible hoursAll Care Therapies
San Francisco, CA2 days ago- ...for this role. Mercor is seeking a Hindi Audio Generalist Evaluator Expert to contribute to a high-impact audio AI research project... ...and evaluation tasks that help train and benchmark advanced language models. This is a short-term, structured engagement ideal for...Temporary workPart time10 hours per week
Mercor
San Francisco, CA6 days ago - ...Francisco, California. The Role: As a Research Engineer - Language Model Pre-Training , you'll shape our language model roadmap... ...our pretraining stack Dataset collection, processing, and evaluation Architecture and methodology research, including optimizer...Work at officeRelocation package
- ...Partnership, Agency & Self-Determination, and Joy. Job Summary The Bilingual Speech-Language Pathologist provides a full range of speech and language... ...special education assessment, planning, instruction, and evaluation Knowledge of child cognitive development and different...BilingualWork at officeRelocation package
Aspire Public Schools
San Francisco, CA1 day ago $58 - $63 per hour
...Research Intern, Model Shaping (Fall 2026) San Francisco About The Role As a... ...optimization methods) Robust and reliable evaluation of foundation model capabilities... ...ideas across machine learning, natural language processing, and ML systems. Past research...Hourly payInternshipWork at officeTogether AI
San Francisco, CA4 days ago$30 - $46 per hour
...Speech Language Pathology Assistant (SLPA) - Remote All Care Therapies is a leading provider of Physical, Occupational, Speech, and... ..., behavioral modifications, adults, & AAC, preferred Bilingual and able to treat in Spanish, preferred Hours and Location...BilingualRemote jobHourly payPart timePrivate practiceLocal areaWork from homeFlexible hours$89k - $105.7k
...Schools. We offer a Speech Language Pathology team lead that can... ...practices for special education. Bilingual (Spanish or Vietnamese)... ...Ability to perform diagnostic evaluations according to CA law and ASHA... ...literacy achievement and to model scaffolding strategies. Support...BilingualFull timeTemporary workWork at office$72.94k - $137.95k
High School World Language Filipino Teacher Salary Range: $72,943-$137,945 annually Length of Appointment: 2026-2027 School Year Calendar... ...quality instruction and equitable support to Filipino-English bilingual students in a diverse, urban setting. Key Qualifications...BilingualFull timeInternshipFlexible hours$70.3k - $76.76k
...serve. The Position The Speech and Language Pathology Assistant will work under the... ...elementary school students in screening, evaluating, diagnosing, and treating disorders of... ...the State of California (required). Bilingual in English and Spanish or Vietnamese (strongly...BilingualLocal area- Mercor is looking for a Real estate / hospitality / events Evaluator to evaluate AI-generated artifacts against quality rubrics. The role... ...while working remotely and meeting deadlines. Strong English language skills are essential, and an advanced degree is preferred. #J...Remote jobContract workWork at office
- Obsidian is seeking a Spanish Audio Generalist Evaluator Expert to contribute to a high-impact audio AI research project. You will handle... ..., and evaluation tasks to help train and benchmark advanced language models. The ideal candidate should have strong writing skills,...Part time10 hours per week
- ...speakers with knowledge of Saudi Arabian dialects to evaluate AI-generated text and voice snippets. This role requires assessing how authentically the models capture Arabic speech, influencing the way AI understands human language. You will analyze audio clips and provide...
$72.94k - $137.95k
Job Class: 0823 Evaluator Salary Range: $72,943 - $137,945 Calendar: Full-Time, School-Term Responsibilities Provide support in communicating... ...of Common Core State Standards for Mathematics and English Language Arts. Knowledge of the role of assessments within pedagogical...Full timeWork at office$144k - $187k
...and frameworks that power MSCI's quantitative risk and factor model research. The team develops the systems and environments that enable... ...participants across the financial ecosystem with a common language. Our research-based data, analytics and indexes, supported by advanced...Flexible hoursMSCI
San Francisco, CA3 days ago$301.75k - $355k
Crusoe Energy Systems LLC is seeking a Senior Director to lead the Model LifeCycle team. This pivotal role focuses on establishing a... ...application development in Machine Learning, particularly Large Language Models. Candidates should possess an advanced degree, with over...$301.75k - $355k
...This Role The Senior Director for the Model LifeCycle team will undertake a pivotal... ...Machine Learning models, including Large Language Models (LLMs). What You’ll Be Working... ...management: versioning, lineage, evaluation, and reproducible fine‑tuning at scale....Temporary work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Bilingual Language Model Evaluator. Be the first to apply!
Related searches
- evaluator San Francisco, CA
- social media evaluator San Francisco, CA
- work from home web search evaluator San Francisco, CA
- quality evaluator San Francisco, CA
- bilingual customer service manager San Francisco, CA
- bilingual customer service representative spanish english San Francisco, CA
- bilingual customer support specialist San Francisco, CA
- bilingual account manager San Francisco, CA
- bilingual work from home San Francisco, CA
- bilingual case manager San Francisco, CA