DevOps Engineer - AI Model Evaluator - AI Trainer
$400 per monthMercor Inc
About the Role Mercor is partnering with a leading AI research lab to support a Frontier Code Agents project. Contributors help evaluate and improve frontier AI coding models through structured technical assessments. The work focuses on realistic infrastructure engineering workflows and model evaluation. Spots are limited and filling quickly on a first come, first serve basis. What You'll Do Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks. Review model-generated implementations involving cloud platforms, Kubernetes, CI/CD systems, observability, and infrastructure automation. Identify bugs, edge cases, reliability issues, and failure modes. Compare outputs from multiple frontier models and assess their strengths and weaknesses. Apply professional engineering judgment to realistic infrastructure engineering scenarios. Time Commitment Sprint based project that runs in 12-24 hour stretches based on client requirement. Compensation $400 per accepted task. Typical tasks take approximately 2–3 hours after ramp-up. Compensation is tied to accepted work. Who Should Apply 2+ years of professional DevOps, SRE, or Cloud Engineering experience. Experience with AWS, Azure, GCP, Kubernetes, Terraform, CI/CD pipelines, or observability tooling. Regular use of AI coding agents such as Cursor, Claude Code, Codex, Windsurf, Gemini CLI, or similar tools. Ability to evaluate model-generated infrastructure and reliability engineering solutions. Experience supporting production-scale systems is preferred. #J-18808-Ljbffr Mercor Inc
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator - AI Trainer in San Francisco, CA vacancy
$85 per hour
...technical talent with leading AI research labs. Headquartered... ...Dorsey . Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)... ...coding agents to complete and evaluate complex infrastructure engineering tasks. Review model-generated implementations involving...SuggestedContract workSummer workRemote work- Mercor is partnering with a leading AI research lab to support a Frontier Code Agents project. You will help evaluate frontier AI coding models by performing infrastructure engineering tasks and reviewing model-generated implementations on cloud platforms, Kubernetes,...Suggested
$85 per hour
...technical talent with leading AI research labs. Headquartered... ...Dorsey . Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)... ...coding agents to complete and evaluate complex infrastructure engineering tasks. Review model-generated implementations involving...SuggestedContract workSummer workRemote work$50 - $75 per hour
A leading tech company based in Australia is seeking an AI Model Evaluator on a contract basis. The role involves evaluating AI-generated responses, writing prompts, and providing justifications based on specific criteria. Ideal candidates will hold a Master's degree in...SuggestedHourly payContract work$70 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...based on sourced input to challenge AI models . Build grading criteria to define correct... ...PhD in materials science, materials engineering, applied physics, chemistry, chemical engineering...SuggestedContract workSummer workImmediate startRemote workMercor
San Francisco, CA4 days ago$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Incident management / reliability / SRE Evaluator Type: Contract Compensation: $... ...asynchronously to meet deadlines and enhance model performance. Qualifications Must-Have...Contract workSummer workWork at officeRemote work$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...customer research and feedback synthesis Evaluator Type: Contract Compensation: $... ...asynchronously to meet deadlines while improving AI model performance . Utilize Microsoft...Contract workSummer workWork at officeRemote work$90 per hour
...technical talent with leading AI research labs. Headquartered in... .... Position: Frontend Engineer — Web Replication Preference Rater... ...fidelity in detail, including box model, typography, color, image... ...that merely look correct. Evaluate responsive behavior and semantic...Contract workSummer workLocal areaRemote work$218.5k - $288k
...Scientist specializing in Small Language Models and AI Training, you will lead research and... .... You will work closely with research, engineering, and product teams to advance model training... ...language models.Design, implement, and evaluate model training experiments to improve...Work at officeFlexible hours3 days per week$180k - $225k
As a Software Engineer on the ML Infrastructure team, you will design... ...engineers to integrate and optimize models for production and research... ...to ensure a fair and thorough evaluation of all applicants.About Us:At... ...is to develop reliable AI systems for the world's most important...Full time$144k - $187k
...power MSCI's quantitative risk and factor model research. The team develops the systems... ...research infrastructure that integrates AI to accelerate model development, validation... ...Collaborating cross-functionally with research, engineering, and data teams Presenting complex...Flexible hoursMSCI
San Francisco, CA2 days ago$15 - $20 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...tools . Generate high-quality human evaluation data by identifying response strengths, areas... ...and completeness of responses. Ensure model responses align with expected conversational...Contract workSummer workRemote work$400 per month
...Mercor is partnering with a leading AI research lab to support a Frontier... ...project. Contributors help evaluate and improve frontier AI coding models through structured technical assessments... ...work focuses on realistic data engineering workflows and model evaluation. Spots...$150 per hour
A leading AI data collection platform is seeking AI Trainers - Machine Learning Specialists to assist in training and evaluating advanced AI models. Successful candidates will complete training tasks, provide expert feedback, and help enhance AI's capabilities. This role...Remote jobWork from home$30 per hour
A leading platform for AI training is seeking advanced Japanese speakers to join as Domain... ...complete various AI training tasks and evaluate AI responses in Japanese. Compensation is... ...a dynamic environment aimed at improving powerful AI models. #J-18808-Ljbffr ProlificRemote jobHourly payWork from homeFlexible hours$40 per hour
Prolific is seeking AI Trainers with advanced SQL development skills to train and evaluate AI models. The role requires strong attention to detail, ability to work on complex tasks, and a reliable internet connection. Successful candidates will join Prolific as Domain...Remote jobFlexible hours- Armada is seeking a patient, articulate Part-Time Technical Trainer to deliver hands-on training and create demo content for our GPUaaS platform. This role combines technical enablement with customer-facing teaching, traveling to sites worldwide when required, and producing...Part timeRemote workWorldwide
- Mercor, in collaboration with a leading AI lab, seeks experienced FP&A and treasury professionals to translate real planning and cash management work into structured training data for AI that reasons like finance teams. You will design scenarios such as budgets, rolling...
$30 per hour
AI Trainer - Advanced Japanese Fluency Prolific is building the biggest pool of quality human data in the world. Over 35,00... ...looking for advanced Japanese speakers to help train and evaluate cutting‑edge AI models. If you have the necessary experience, you’ll take a...Remote jobHourly paySelf employmentWork from homeFlexible hours- Obsidian is partnering with an AI research initiative to support a Frontier Code Agents project in San Francisco. You will evaluate frontier AI coding models by performing realistic data engineering tasks, reviewing ETL pipelines, data warehouses, and distributed systems...
- A technology consulting firm in California is seeking a Senior DevOps Engineer to enhance cloud-based data management platforms. The ideal candidate will have strong AWS and CI/CD skills, alongside expertise in tools like Databricks and Collibra. Duties include defining...Contract work
$150k - $200k
...Get AI-powered advice on this job and more exclusive features. Understanding Recruitment provided pay range This range is provided... ...working with a leading AI company, who are looking for a DevOps Engineer to join their ever-expanding Engineering team who can bridge the...Full timeWork at officeImmediate start2 days per week3 days per weekUnderstanding Recruitment
San Francisco, CA4 days ago- A leading data and AI company in San Francisco is seeking a Senior Engineer to enhance their Model Serving platform. This role requires expertise in building large-scale distributed systems and collaboration across teams to optimize performance and reliability. Ideal candidates...
$130k - $196.5k
...service environment creation for engineering teams.Lead and collaborate... ...quality.Mentor engineers on DevOps best practices, helping teams... ...infrastructure technologies by evaluating options, building proofs of... ...processing tools, analytics, and AI/ML workloads across LiveRamp...Full timeWork from homeFlexible hoursNight shift- ...author and review high-quality academic assessment content for an AI research initiative. You will write and verify rigorous multiple... ...questions across core history and political science domains, evaluate solution quality, and help establish gold-standard benchmarks used...Remote work
- ...Welo Data is seeking Data Labeling Associates in California to evaluate AI outputs and ensure cultural context and safety in Arabic datasets. This role requires professional-level proficiency in Portuguese (Brazil), a bachelor's degree, and at least 2 years of experience...
Welo Data
San Francisco, CA20 hours ago $180k - $275k
...Base pay range $180,000.00/yr – $275,000.00/yr We’re building an AI-native fintech platform that’s modernizing how finance teams... ...can support complex financial workflows at speed. Position DevOps Engineer – Own and evolve the cloud infrastructure that underpins our product...Full timeStrativ Group
San Francisco, CA20 hours ago$150 per hour
A leading AI data firm is seeking qualified Lawyers to serve as AI Trainers. In this remote role, you will guide AI models using your legal expertise by completing tasks such as analyzing legal documents and providing feedback. The position offers flexible hours and competitive...Remote jobHourly payFlexible hours- ...located in San Francisco is seeking an innovative Quality Engineer for their AI products. This role blends ops, strategy, and analytics to... ...leading labs, and ensure user satisfaction through effective evaluation baselines. Competitive salary and benefits offered, with a...
$201k
A leading technology firm in San Francisco seeks a Senior Software Engineer for the Model LifeCycle team. This role focuses on building a managed platform for application development, specifically leveraging Machine Learning models, including LLMs. Candidates should have...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator - AI Trainer. Be the first to apply!
Related searches
- senior devops cloud engineer San Francisco, CA
- senior devops engineer San Francisco, CA
- devops engineer sre San Francisco, CA
- devops cloud engineer San Francisco, CA
- devops engineer San Francisco, CA
- devops engineer remote San Francisco, CA
- senior devops engineer remote San Francisco, CA
- devops aws developer (remote) San Francisco, CA
- devops engineer full time San Francisco, CA
- big data devops engineer San Francisco, CA