DevOps Engineer - AI Model Evaluator - AI Trainer
$85 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
- Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
- Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , and infrastructure automation .
- Identify bugs, edge cases, reliability issues, and failure modes in model outputs.
- Compare outputs from multiple frontier models to assess strengths and weaknesses.
- Apply professional engineering judgment to realistic infrastructure engineering scenarios.
Qualifications
Must-Have
- 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
- Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling.
- Regular use of AI coding agents like Cursor , Claude Code , Codex , Windsurf , Gemini CLI , or similar tools.
- Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
- Experience supporting production-scale systems .
Compensation & Legal
- $400 per accepted task .
- Compensation is tied to accepted work.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on us.fitly.work
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
#hiringmercorVacancy posted a month ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator - AI Trainer in San Francisco, CA vacancy
$85 per hour
...technical talent with leading AI research labs. Headquartered... ...Dorsey . Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)... ...coding agents to complete and evaluate complex infrastructure engineering tasks. Review model-generated implementations involving...SuggestedContract workSummer workRemote work$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Incident management / reliability / SRE Evaluator Type: Contract Compensation: $... ...asynchronously to meet deadlines and enhance model performance. Qualifications Must-Have...SuggestedContract workSummer workWork at officeRemote work$218.5k - $288k
...Scientist specializing in Small Language Models and AI Training, you will lead research and... .... You will work closely with research, engineering, and product teams to advance model training... ...language models.Design, implement, and evaluate model training experiments to improve...SuggestedWork at officeFlexible hours3 days per week$180k - $225k
As a Software Engineer on the ML Infrastructure team, you will design... ...engineers to integrate and optimize models for production and research... ...to ensure a fair and thorough evaluation of all applicants.About Us:At... ...is to develop reliable AI systems for the world's most important...SuggestedFull time$144k - $187k
...power MSCI's quantitative risk and factor model research. The team develops the systems... ...research infrastructure that integrates AI to accelerate model development, validation... ...Collaborating cross-functionally with research, engineering, and data teams Presenting complex...SuggestedFlexible hoursMSCI
San Francisco, CA3 days ago- ...ll apply your expertise to help train next-generation AI systems. Your work will shape how models learn, reason, and perform through high-quality, real... ...Experience with quality assurance and rubric-based evaluation of legal documents is a plus. Demonstrated commitment...For contractorsRemote work
YO AI Labs
San Francisco, CA10 days ago $15 - $20 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...tools . Generate high-quality human evaluation data by identifying response strengths, areas... ...and completeness of responses. Ensure model responses align with expected conversational...Contract workSummer workRemote work- Armada is seeking a patient, articulate Part-Time Technical Trainer to deliver hands-on training and create demo content for our GPUaaS platform. This role combines technical enablement with customer-facing teaching, traveling to sites worldwide when required, and producing...Part timeRemote workWorldwide
- ...Outlier connect subject matter experts with AI training projects Outlier is one of the world’s largest expert platforms, connecting... ...700,000 MAs, PhDs, and college graduates worldwide to improve AI models by providing expert human feedback. Experts on Outlier work wherever...Weekly payWorldwide
Outlier AI
San Francisco, CA15 hours ago $30 per hour
A leading platform for AI training is seeking advanced Japanese speakers to join as Domain... ...complete various AI training tasks and evaluate AI responses in Japanese. Compensation is... ...a dynamic environment aimed at improving powerful AI models. #J-18808-Ljbffr ProlificRemote jobHourly payWork from homeFlexible hours$30 per hour
AI Trainer - Advanced Japanese Fluency Prolific is building the biggest pool of quality human data in the world. Over 35,00... ...looking for advanced Japanese speakers to help train and evaluate cutting‑edge AI models. If you have the necessary experience, you’ll take a...Remote jobHourly paySelf employmentWork from homeFlexible hours$170k - $216k
...of miles of driving data from a diverse set of sensors, enabling engineers like you to (1) develop methods for efficiently and continuously learning from large scale real-world data, to (2) develop models and model training at scale, to (3) analyze real-world behavior and...Full timeRemote workWaymo
San Francisco, CA15 hours ago$195k - $250k
...consulting rooted in data, technology, and AI, we help clients drive value—supporting... ...and You We are hiring a Cloud DevOps Engineer to support a growing portfolio of AI development... ...compensation packages are determined by evaluating a wide array of factors unique to each...Full timeWork at officeLocal areaRemote work2 days per weekAccordion
San Francisco, CA1 day ago- ...Description Job Description We are looking for a Senior Software Engineer, Infrastructure with ideally 5+ years of experience to own and... ...build, release, and runtime foundations at a fast-scaling voice AI company. You'll design and automate deployment pipelines across...
NxT Level
San Francisco, CAa month ago $150k - $200k
...Description Job Description About the Role We're a venture-backed AI messaging infrastructure startup building production-grade... ...across popular messaging platforms. As a founding macOS DevOps Engineer (mid-level Member of Technical Staff), you'll be a core member...Remote workClera
San Francisco, CA8 days ago$176.6k - $239k
...Solutions Architect, you will partner some of the world’s leading AI model providers companies, AWS Sales, and several other AWS teams to... ...areas (e.g. software development, cloud computing, systems engineering, infrastructure, security, networking, data & analytics)...Local areaWorldwideFlexible hours- ...or Kubernetes-based environments. Experience building AI-powered solutions, MCP Servers, Agentic AI systems, or... ...skills. Preferred Qualifications Experience with DevOps and Site Reliability Engineering (SRE) practices. Strong production support, incident...
Eitacies Inc
San Francisco, CAa month ago $201k
A leading technology firm in San Francisco seeks a Senior Software Engineer for the Model LifeCycle team. This role focuses on building a managed platform for application development, specifically leveraging Machine Learning models, including LLMs. Candidates should have...- ...critical inference for the world's most dynamic AI companies, like Cursor, Notion,... ...the frontier of AI to bring cutting-edge models into production. We're growing quickly and... ...Conviction. Join us and help build the platform engineers turn to to ship AI products. THE...Full timeFlexible hours
Baseten
San Francisco, CA15 hours ago $172.5k - $260.1k
...DetailsAbout SalesforceSalesforce is the #1 AI CRM, where humans with agents drive... ....Lead Member of Technical Staff, Release Engineering, Agentic Platform (Java / Node.js / iOS)We... ...tools to help our recruiters assess and evaluate candidates’ resumes and qualifications throughout...Full time- ...enterprise and developers alike to use and access our start-of-the-art AI models, allowing them to do things that they’ve never been able to... ...via model inference. About the Role We are looking for an engineer who wants to take the world's largest and most capable AI...Full time
OpenAI
San Francisco, CA15 hours ago - ...About the Team We’re hiring software engineers to make OpenAI’s Model Performance teams more productive. These teams work on the systems, tooling,... ...with forward progress About OpenAI OpenAI is an AI research and deployment company dedicated to ensuring that...Full time
OpenAI
San Francisco, CA15 hours ago $298k - $368k
...of miles of driving data from a diverse set of sensors, enabling engineers like you to (1) develop methods for efficiently and continuously learning from large scale real-world data, to (2) develop models and model training at scale, to (3) analyze real-world behavior and...Full timeRemote workWaymo
San Francisco, CA15 hours ago$130k - $196.5k
...service environment creation for engineering teams.Lead and collaborate... ...quality.Mentor engineers on DevOps best practices, helping teams... ...infrastructure technologies by evaluating options, building proofs of... ...processing tools, analytics, and AI/ML workloads across LiveRamp...Full timeWork from homeFlexible hoursNight shift- ...We are rebuilding biotech for the AI era. When a breakthrough is delayed, the world... ...structured data, and run AI agents and models directly in their workflows. Over 200,000... ...-saving therapeutics. As a full-stack engineer on the team, you’ll focus on building the...Full timeWork at officeLocal areaMonday to FridayShift work
Benchling
San Francisco, CA15 hours ago $180k - $275k
...Base pay range $180,000.00/yr – $275,000.00/yr We’re building an AI-native fintech platform that’s modernizing how finance teams... ...can support complex financial workflows at speed. Position DevOps Engineer – Own and evolve the cloud infrastructure that underpins our product...Full timeStrativ Group
San Francisco, CA1 day ago$146.5k - $190.5k
...strategy, design, and delivery of security engineering solutions that protect the company's... .... You will manage a team of security and DevOps engineers, driving a culture of security-... ...stakeholders, and executives alike.Familiarity with AI and machine learning capabilities as...Minimum wage$153k - $202k
...Join to apply for the DevOps Engineer role at Merge Merge is the leading provider of agentic tools and customer-facing integrations for frontier... ...with a single API, and Merge Agent Handler, which empowers AI agents with secure access to thousands of third-party tools....Full timeWork at officeHome officeMerge LLC
San Francisco, CA1 day ago$30 per hour
...AI Trainer – Visual & Graphic Design Expert (Remote) About Prolific Prolific is not just another player in the AI space... ...Visual Designers to act as Domain Experts for a high‑level AI evaluation project. AI models are evolving beyond simple image generation and now...Remote workWork from homeFlexible hoursProlific
San Francisco, CA1 day ago$238k - $302k
...in simulation across 15+ U.S. states. The Large Model Evaluation team is at the nexus of Waymo’s AI ambition . With advancements in Large Language Models... ...critical. We are looking for quantitatively-minded engineers to research and propose new ways to assess the ML...Full timeRemote workWaymo
San Francisco, CA15 hours ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator - AI Trainer. Be the first to apply!
Related searches
- big data devops engineer San Francisco, CA
- devops engineer sre San Francisco, CA
- devops engineer remote San Francisco, CA
- senior devops engineer remote San Francisco, CA
- senior devops engineer San Francisco, CA
- devops engineer San Francisco, CA
- senior devops cloud engineer San Francisco, CA
- devops engineer full time San Francisco, CA
- devops aws developer (remote) San Francisco, CA
- work from home web search evaluator San Francisco, CA