DevOps Engineer - AI Model Evaluator
$85 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
- Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
- Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , and infrastructure automation .
- Identify bugs, edge cases, reliability issues, and failure modes in model outputs.
- Compare outputs from multiple frontier models to assess strengths and weaknesses.
- Apply professional engineering judgment to realistic infrastructure engineering scenarios.
Qualifications
Must-Have
- 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
- Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling.
- Regular use of AI coding agents like Cursor , Claude Code , Codex , Windsurf , Gemini CLI , or similar tools.
- Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
- Experience supporting production-scale systems .
Compensation & Legal
- $400 per accepted task .
- Compensation is tied to accepted work.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on us.fitly.work
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted a month ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator in San Francisco, CA vacancy
$85 per hour
...technical talent with leading AI research labs. Headquartered... ...Dorsey . Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)... ...coding agents to complete and evaluate complex infrastructure engineering tasks. Review model-generated implementations involving...SuggestedContract workSummer workRemote work$50 - $75 per hour
A leading tech company based in Australia is seeking an AI Model Evaluator on a contract basis. The role involves evaluating AI-generated responses, writing prompts, and providing justifications based on specific criteria. Ideal candidates will hold a Master's degree in...SuggestedHourly payContract work- ...Artificial Intelligence (DSAI) group within Model Risk Management (MRM). The... ...third parties with machine learning and AI technologies primarily for the purpose of... ...discipline such as mathematics, statistics, engineering, physics, economics, or computer scienceDesired...SuggestedFull timeWork experience placementVisa sponsorship
$218.5k - $288k
...Scientist specializing in Small Language Models and AI Training, you will lead research and... .... You will work closely with research, engineering, and product teams to advance model training... ...language models.Design, implement, and evaluate model training experiments to improve...SuggestedWork at officeFlexible hours3 days per week$180k - $225k
As a Software Engineer on the ML Infrastructure team, you will design... ...engineers to integrate and optimize models for production and research... ...to ensure a fair and thorough evaluation of all applicants.About Us:At... ...is to develop reliable AI systems for the world's most important...SuggestedFull time- ...Description Job Description We are looking for a Senior Software Engineer, Infrastructure with ideally 5+ years of experience to own and... ...build, release, and runtime foundations at a fast-scaling voice AI company. You'll design and automate deployment pipelines across...
NxT Level
San Francisco, CAmore than 2 months ago $150k - $240k
...DevOps Engineer Title of Role: DevOps Engineer Location: San Francisco, on-site or remote Company Stage of Funding: Series B Office... ...software development space, particularly within API SDKs and AI-driven solutions. With a strong commitment to innovation and...Work at officeRemote workRecruiting from Scratch
San Francisco, CA3 days ago$130k - $196.5k
...service environment creation for engineering teams.Lead and collaborate... ...quality.Mentor engineers on DevOps best practices, helping teams... ...infrastructure technologies by evaluating options, building proofs of... ...processing tools, analytics, and AI/ML workloads across LiveRamp...Full timeWork from homeFlexible hoursNight shift$176.6k - $239k
...Solutions Architect, you will partner some of the world’s leading AI model providers companies, AWS Sales, and several other AWS teams to... ...areas (e.g. software development, cloud computing, systems engineering, infrastructure, security, networking, data & analytics)...Local areaWorldwideFlexible hours$172.5k - $260.1k
...DetailsAbout SalesforceSalesforce is the #1 AI CRM, where humans with agents drive... ....Lead Member of Technical Staff, Release Engineering, Agentic Platform (Java / Node.js / iOS)We... ...tools to help our recruiters assess and evaluate candidates’ resumes and qualifications throughout...Full time- ...or Kubernetes-based environments. Experience building AI-powered solutions, MCP Servers, Agentic AI systems, or... ...skills. Preferred Qualifications Experience with DevOps and Site Reliability Engineering (SRE) practices. Strong production support, incident...
Eitacies Inc
San Francisco, CAa month ago - ...critical inference for the world's most dynamic AI companies, like Cursor, Notion,... ...the frontier of AI to bring cutting-edge models into production. We're growing quickly and... ...Conviction. Join us and help build the platform engineers turn to to ship AI products. THE...Full timeFlexible hours
Baseten
San Francisco, CA16 hours ago - ...We are rebuilding biotech for the AI era. When a breakthrough is delayed, the world... ...structured data, and run AI agents and models directly in their workflows. Over 200,000... ...-saving therapeutics. As a full-stack engineer on the team, you’ll focus on building the...Full timeWork at officeLocal areaMonday to FridayShift work
Benchling
San Francisco, CA16 hours ago - ...About the Team We’re hiring software engineers to make OpenAI’s Model Performance teams more productive. These teams work on the systems, tooling,... ...with forward progress About OpenAI OpenAI is an AI research and deployment company dedicated to ensuring that...Full time
OpenAI
San Francisco, CA16 hours ago - ...enterprise and developers alike to use and access our start-of-the-art AI models, allowing them to do things that they’ve never been able to... ...via model inference. About the Role We are looking for an engineer who wants to take the world's largest and most capable AI...Full time
OpenAI
San Francisco, CA16 hours ago - ...is taking the hard out of hardware, by developing the first AI Hardware Engineer. Our goal is to democratize the ability to create bleeding edge... ...that means everything around it has to just work. As a DevOps Engineer, you'll work on the full-stack systems that power...Local areaShift work
Flux Defunct
San Francisco, CA16 hours ago $180k - $275k
...Base pay range $180,000.00/yr – $275,000.00/yr We’re building an AI-native fintech platform that’s modernizing how finance teams... ...can support complex financial workflows at speed. Position DevOps Engineer – Own and evolve the cloud infrastructure that underpins our product...Full timeStrativ Group
San Francisco, CA1 day ago- ...Summary: This position is responsible for: Build DevOps solutions across applications, network, deployment & scaling to... ...~ Experience scaling different geographies. High level of AI usage in Day-to-Day activity. (Good to have) Experience...Work experience placementWork at office
Texas State Library and Archives Commision
San Francisco, CA2 days ago - Cincinnatus LLC is placing finance SMEs at a leading AI lab to critically evaluate AI model outputs for financial tasks. The role focuses on rigorous, rubric-based assessment of model performance and constructing finance-focused evaluation frameworks. Candidates should...Weekday work
$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...: Data analysis / quantitative readouts Evaluator Type: Contract Compensation: $... ...asynchronously to meet deadlines and improve AI model performance . Qualifications Must-...Contract workSummer workWork at officeRemote workMercor
San Francisco, CA3 days ago- ...DevOps Engineer Encord is the universal data layer for AI that helps 300+ AI teams train and run models on the right data. Our platform indexes, curates, annotates, and evaluates data across the full AI lifecycle, from development through production. We're looking...Work at officeFlexible hours
Encord
San Francisco, CA16 hours ago - Vapi Inc. in San Francisco is seeking a dedicated engineer to enhance their deploy pipeline and implement progressive delivery systems. The... ...on customer satisfaction for internal engineers. Join a dynamic environment driving innovation in voice AI! #J-18808-Ljbffr VAPI
- ...investors, we're building the category-defining AI workflow automation platform that... ...Role Plenful is hiring a Senior DevOps Engineer to join our engineering team and help build... ...and New York. R&D roles follow a hybrid model, with two days per week in our San...Full timeWork at officeLocal areaRemote workFlexible hours2 days per week
Plenful
San Francisco, CA22 hours ago - A pioneering technology company in San Francisco is seeking a DevOps Engineer to join the founding team. This role requires building and maintaining scalable cloud infrastructure, own CI/CD systems, and implement observability tooling. Ideal candidates have 4-10+ years...
- A leading data and AI company in San Francisco is seeking a Staff Engineer to design and implement systems for their AI/ML Model Serving platform. You will collaborate with product, infrastructure, and research teams to ensure high-performance system delivery. The ideal...
- Synthires is offering a part-time role for PhD-level Chemistry experts to contribute to AI safety and evaluation projects. The work involves applying scientific expertise to understand and improve how AI systems handle specialized chemistry topics, with training provided...Remote jobPart time
$172.43k - $230.95k
...intelligence. As the only vertically integrated AI infrastructure company built from the... ....About This Role:The Senior Software Engineer for the AI Model Lifecycle team will play a crucial... ...management: versioning, lineage, evaluation, and reproducible fine-tuning at scale...Temporary work- Obsidian is looking for expert Evaluators in Finance operations/audit support to review AI-generated work products for accuracy and quality. This remote hourly position requires a minimum of 5 years in finance and fluency in English. Your role will involve evaluating outputs...Remote jobHourly payWork at office
$80 - $120 per hour
Mercor, based in San Francisco, is seeking a Cybersecurity / IT GRC Evaluator to evaluate AI-generated artifacts and provide structured feedback. Candidates should have over 5 years of relevant experience and fluency in English. This remote position offers a competitive...Remote jobHourly payWork at office- Obsidian is seeking a Spanish Audio Generalist Evaluator Expert to contribute to a high-impact audio AI research project. You will handle transcription, annotation... ...to help train and benchmark advanced language models. The ideal candidate should have strong writing skills...Part time10 hours per week
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator. Be the first to apply!
Related searches
- big data devops engineer San Francisco, CA
- devops cloud engineer San Francisco, CA
- devops engineer sre San Francisco, CA
- devops engineer remote San Francisco, CA
- senior devops engineer remote San Francisco, CA
- senior devops engineer San Francisco, CA
- devops engineer San Francisco, CA
- senior devops cloud engineer San Francisco, CA
- devops engineer full time San Francisco, CA
- devops aws developer (remote) San Francisco, CA