Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

DevOps Engineer - AI Model Evaluator - AI Trainer

$400 per month

Mercor Inc

About the Role Mercor is partnering with a leading AI research lab to support a Frontier Code Agents project. Contributors help evaluate and improve frontier AI coding models through structured technical assessments. The work focuses on realistic infrastructure engineering workflows and model evaluation. Spots are limited and filling quickly on a first come, first serve basis. What You'll Do Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks. Review model-generated implementations involving cloud platforms, Kubernetes, CI/CD systems, observability, and infrastructure automation. Identify bugs, edge cases, reliability issues, and failure modes. Compare outputs from multiple frontier models and assess their strengths and weaknesses. Apply professional engineering judgment to realistic infrastructure engineering scenarios. Time Commitment Sprint based project that runs in 12-24 hour stretches based on client requirement. Compensation $400 per accepted task. Typical tasks take approximately 2–3 hours after ramp-up. Compensation is tied to accepted work. Who Should Apply 2+ years of professional DevOps, SRE, or Cloud Engineering experience. Experience with AWS, Azure, GCP, Kubernetes, Terraform, CI/CD pipelines, or observability tooling. Regular use of AI coding agents such as Cursor, Claude Code, Codex, Windsurf, Gemini CLI, or similar tools. Ability to evaluate model-generated infrastructure and reliability engineering solutions. Experience supporting production-scale systems is preferred. #J-18808-Ljbffr Mercor Inc

Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator - AI Trainer in San Francisco, CA vacancy
  • $10 - $20 per hour

     ...Role Overview Help train next-generation robotic AI systems through precise, hands-on data collection in a San Francisco lab. You will...  ...everyday manipulation tasks that support how robot learning models learn, reason, and perform. No prior robotics or AI experience is... 
    Suggested
    Hourly pay
    Contract work

    SaidGig

    San Francisco, CA
    4 days ago
  • $30 per hour

    A leading platform for AI training is seeking advanced Japanese speakers to join as Domain...  ...complete various AI training tasks and evaluate AI responses in Japanese. Compensation is...  ...a dynamic environment aimed at improving powerful AI models. #J-18808-Ljbffr Prolific
    Suggested
    Remote job
    Hourly pay
    Work from home
    Flexible hours

    Prolific

    San Francisco, CA
    3 days ago
  • $30 per hour

    AI Trainer - Advanced Japanese Fluency Prolific is building the biggest pool of quality human data in the world. Over 35,00...  ...looking for advanced Japanese speakers to help train and evaluate cutting‑edge AI models. If you have the necessary experience, you’ll take a... 
    Suggested
    Remote job
    Hourly pay
    Self employment
    Work from home
    Flexible hours

    Prolific

    San Francisco, CA
    6 days ago
  • $170k - $216k

     ...of miles of driving data from a diverse set of sensors, enabling engineers like you to (1) develop methods for efficiently and continuously learning from large scale real-world data, to (2) develop models and model training at scale, to (3) analyze real-world behavior and... 
    Suggested
    Full time
    Remote work

    Waymo

    San Francisco, CA
    9 hours ago
  •  ...critical inference for the world's most dynamic AI companies, like Cursor, Notion,...  ...the frontier of AI to bring cutting-edge models into production. We're growing quickly and...  ...Conviction. Join us and help build the platform engineers turn to to ship AI products. THE... 
    Suggested
    Full time
    Flexible hours

    Baseten

    San Francisco, CA
    9 hours ago
  •  ...enterprise and developers alike to use and access our start-of-the-art AI models, allowing them to do things that they’ve never been able to...  ...model inference.  About the Role We are looking for an engineer who wants to take the world's largest and most capable AI... 
    Full time

    OpenAI

    San Francisco, CA
    9 hours ago
  • $298k - $368k

     ...of miles of driving data from a diverse set of sensors, enabling engineers like you to (1) develop methods for efficiently and continuously learning from large scale real-world data, to (2) develop models and model training at scale, to (3) analyze real-world behavior and... 
    Full time
    Remote work

    Waymo

    San Francisco, CA
    9 hours ago
  •  ...About the Team We’re hiring software engineers to make OpenAI’s Model Performance teams more productive. These teams work on the systems, tooling,...  ...with forward progress About OpenAI OpenAI is an AI research and deployment company dedicated to ensuring that... 
    Full time

    OpenAI

    San Francisco, CA
    9 hours ago
  •  ...OpenAI’s Hardware organization develops AI-native silicon and system-level solutions...  ...to power the next generation of frontier models. By co-designing chips, systems, tools, and...  ...build the model runtime within the inference engine that executes complex, frontier models at... 
    Full time

    OpenAI

    San Francisco, CA
    9 hours ago
  • Welo Data is seeking Data Labeling Associates in California to evaluate AI outputs and ensure cultural context and safety in Arabic datasets. This role requires professional-level proficiency in Portuguese (Brazil), a bachelor's degree, and at least 2 years of experience... 

    Welo Data

    San Francisco, CA
    3 days ago
  • Obsidian is hiring expert Evaluators in Healthcare operations for a remote, hourly engagement. You will review AI-generated work products for accuracy, rigor, and domain quality, leveraging your extensive subject-matter expertise in the field. The ideal candidate has over... 
    Remote job
    Hourly pay
    Work at office

    Obsidian

    San Francisco, CA
    6 days ago
  •  ...Description Job Description DevOps Engineer Role Type: Contractor...  ...DevOps Engineers to create and evaluate production-style infrastructure...  ...tasks for next-generation AI systems. You will work on CI/...  ...Previous experience with AI training data or model evaluation.... 
    For contractors
    Remote work

    YO AI Labs

    San Francisco, CA
    22 days ago
  • $160k - $250k

     ...creating cutting-edge AI/ML products that solve...  ...for a seasoned DevSecOps engineer with significant experience...  ...SLA Service cost modeling and ongoing cost...  ...Configurations, etc. ~ Solid DevOps experience, preferably...  ...applicable laws. We evaluate candidates holistically... 
    Work at office
    Local area
    Flexible hours

    Ellipsis Health

    San Francisco, CA
    2 days ago
  •  ...Description Job Description We are looking for a Senior Software Engineer, Infrastructure with ideally 5+ years of experience to own and...  ...build, release, and runtime foundations at a fast-scaling voice AI company. You'll design and automate deployment pipelines across... 

    NxT Level

    San Francisco, CA
    24 days ago
  •  ...Kubernetes-based environments. • Experience building AI-powered solutions, MCP Servers, Agentic AI systems, or GenAI...  .... Preferred Qualifications • Experience with DevOps and Site Reliability Engineering (SRE) practices. • Strong production support, incident... 

    Eitacies Inc

    San Francisco, CA
    24 days ago
  • $150k - $200k

    Get AI-powered advice on this job and more exclusive features. Understanding Recruitment provided pay range This range is provided...  ...working with a leading AI company, who are looking for a DevOps Engineer to join their ever-expanding Engineering team who can bridge the... 
    Full time
    Work at office
    Immediate start
    2 days per week
    3 days per week

    Understanding Recruitment

    San Francisco, CA
    6 days ago
  • $40 per hour

    Overview AI Trainer - Advanced SQL Developers About Prolific Prolific is not just another player in the AI space - we are...  ...role We’re looking for SQL Developers to help train and evaluate cutting-edge AI models. If you have the necessary experience, we’ll send you an... 
    Remote job
    Self employment
    Work from home
    Flexible hours

    Prolific

    San Francisco, CA
    3 days ago
  • $183k - $248k

     ...out of hardware. To do this, we’re building the world's first AI hardware engineer. Founded in 2019, our platform enables anyone to go from...  ...and that means everything around it has to just work. As a DevOps Engineer, you’ll work on the full-stack systems that power Flux... 
    Local area
    Shift work

    Flux

    San Francisco, CA
    22 days ago
  • Senior DevOps Engineer Stack AI is building cutting-edge infrastructure at the intersection of AI and enterprise. Our mission is to enable companies to safely and reliably deploy AI at scale. As we grow, we're looking for a Senior DevOps Engineer who can design, build,... 
    Work at office
    Immediate start
    Remote work

    Stack AI

    San Francisco, CA
    4 days ago
  • $145k - $165k

     ...Description Evolver is seeking a skilled DevOps Engineer who has transitioned into network...  ...Terraform, Azure DevOps pipelines, and AI-assisted development tools such as Claude...  ...of experience building AIOps pipelines, Model Context Protocol (MCP) integrations or LLM... 
    Flexible hours

    Evolver Commercial

    San Francisco, CA
    24 days ago
  • $220k - $320k

     ...trains and hosts specialized language models for companies that need frontier-quality AI at a fraction of the cost. The...  ...-to-end: distillation, training, evaluation, and planet-scale hosting. We...  ...a well-funded ten-person team of engineers who work in-person in downtown... 
    Work at office

    Inference

    San Francisco, CA
    4 days ago
  • $80 - $120 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...Jack Dorsey . Position: Data analysis / quantitative readouts Evaluator Type: Contract Compensation: $80-$120/hour Location: Remote Role... 
    Contract work
    Summer work
    Work at office
    Remote work

    Mercor

    San Francisco, CA
    3 days ago
  • $150k - $180k

     ...is the universal data layer for AI that helps 300+ AI teams train and run models on the right data. Our platform...  ...indexes, curates, annotates, and evaluates data across the full AI lifecycle...  ...The role We're looking for a DevOps Engineer with 4–5 years of experience to... 
    Work at office
    Flexible hours

    Encord

    San Francisco, CA
    24 days ago
  •  ...Join a leading AI lab's cutting-edge GenAI team to be at the...  ...most advanced Large Language Models. 1. Overview Join a leading...  ...re seeking talented Cloud and DevOps Engineers with deep, hands-on expertise...  ...infrastructure automation problems. Evaluate infrastructure engineering... 
    Full time
    Contract work
    Weekday work

    Mercor

    San Francisco, CA
    8 days ago
  • $180k - $275k

     ...pay range $180,000.00/yr - $275,000.00/yr We’re building an AI-native fintech platform that’s modernizing how finance teams operate...  ...can support complex financial workflows at speed. Position DevOps Engineer - Own and evolve the cloud infrastructure that underpins our... 
    Full time

    Strativ Group

    San Francisco, CA
    5 days ago
  • $230k - $320k

    DevOps Engineer At EliseAI, we're improving the industries that matter most: housing and healthcare. Everyone needs a place to live and access...  ...often harder to secure than they should be. By integrating AI agents deeply into existing workflows, we make them more efficient... 
    Work at office
    Local area
    Relocation

    EliseAI

    San Francisco, CA
    2 days ago
  • $153k - $202k

    Join to apply for the DevOps Engineer role at Merge Merge is the leading provider of agentic tools and customer-facing integrations for frontier...  ...with a single API, and Merge Agent Handler, which empowers AI agents with secure access to thousands of third-party tools.... 
    Full time
    Work at office
    Home office

    Merge LLC

    San Francisco, CA
    6 days ago
  •  ...business will answer their phone with voice AI. We are making sure that when they build...  ...high-pressure environment to work in our engineering. This role is critical to the success of...  ...upgrades. Qualifications: 5+ years as a DevOps engineer Experience writing async web apps... 
    Full time
    Work at office

    Phonely Ltd

    San Francisco, CA
    4 days ago
  • $140k - $200k

    DevOps Engineer San Francisco, CA About Pump.co Cloud spend is a whopping $500 billion/yr, the biggest growing expense category for any tech...  ...is building the fastest way to save ~60% on cloud spend. Our AI-powered platform not only fully automates savings, but we also... 
    Work at office

    Pump

    San Francisco, CA
    4 days ago
  • Senior DevOps Engineer Skyfire empowers AI to present verified identities, access essential services and process payments without human intervention. From API account creation to monetizing websites, we drive commerce for the world's fastest-growing consumer base: AI agents... 
    Work at office
    Shift work

    Skyfire

    San Francisco, CA
    4 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator - AI Trainer. Be the first to apply!