DevOps Engineer - AI Model Evaluator
$85 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
- Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
- Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , observability , and infrastructure automation .
- Identify bugs, edge cases, reliability issues, and failure modes in model outputs.
- Compare outputs from multiple frontier models to assess their strengths and weaknesses.
- Apply professional engineering judgment to realistic infrastructure engineering scenarios.
Qualifications
Must-Have
- 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
- Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling.
- Regular use of AI coding agents such as Cursor, Claude Code, Codex, Windsurf, Gemini CLI, or similar tools.
- Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
- Experience supporting production-scale systems.
Compensation & Legal
- $400 per accepted task
- Compensation tied to accepted work.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on us.fitly.work
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 9 days ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator in San Francisco, CA vacancy
$400 per month
...partnering with a leading AI research lab to support a... .... Contributors help evaluate and improve frontier AI coding models through structured technical... ...realistic infrastructure engineering workflows and model evaluation... ...2+ years of professional DevOps, SRE, or Cloud...SuggestedMercor Inc
San Francisco, CA3 days ago- ...is building the world’s leading Agentic AI platform to transform enterprise productivity... ..., Flipkart, and Okta, our team includes engineers from premier tech companies and graduates... ...scalingQualificationsRequired4+ years DevOps, SRE, or platform engineering experienceExpert...SuggestedRemote work
Ema
San Francisco, CA3 days ago $150k - $240k
...DevOps EngineerTitle of Role: DevOps EngineerLocation: San Francisco, on-site or remoteCompany... ...space, particularly within API SDKs and AI-driven solutions. With a strong... ...ElasticSearch, RDS, and Redshift.Collaborate with engineering teams to streamline CI/CD pipelines using...SuggestedRecruiting from Scratch
San Francisco, CA2 days ago$150k - $200k
...Get AI-powered advice on this job and more exclusive features. Understanding Recruitment provided pay range This range is provided... ...working with a leading AI company, who are looking for a DevOps Engineer to join their ever-expanding Engineering team who can bridge the...SuggestedFull timeWork at officeImmediate start2 days per week3 days per weekUnderstanding Recruitment
San Francisco, CA2 days ago- ...Obsidian is hiring expert Evaluators in Real estate, hospitality, and events to review AI-generated work for accuracy, rigor, and domain quality. This remote position requires deep expertise and involves grading outputs like documents and presentations. Applicants must...SuggestedWork at officeRemote work
Obsidian
San Francisco, CA1 day ago - ...is taking the hard out of hardware, by developing the first AI Hardware Engineer. Our goal is to democratize the ability to create bleeding... ...and that means everything around it has to just work. As a DevOps Engineer, you’ll work on the full‑stack systems that power...Shift work
Flux Enterprise
San Francisco, CA3 days ago $130k - $196.5k
...innovators. As consumers embrace new AI-driven experiences, the... ...at LiveRamp.com.As a Senior DevOps Engineer, you will join a platform-... ...frameworks, and Large Language Models (LLMs) into the Software Development... ...audit-ready infrastructure.Evaluate, introduce, and standardize...WorldwideLiveRamp
San Francisco, CA1 day ago$116.2k - $229.1k
Position Summary AI & Engineering/EaaS - DevOps Engineer IIIBuild and scale modern DevOps capabilities that help clients accelerate delivery... ...engineering processes and define target-state DevOps operating models, tooling, and workflowsBuilding and maintaining cloud...Local areaDeloitte
San Francisco, CA1 day ago$131.75k - $178.25k
...professional achievement and career growth. The Senior Infrastructure DevOps Engineer, under the direction of the Senior Manager of Infrastructure,... ...Apps.Experience with managing O365 services.Experience with AI technologies such as Copilot or Claude.Knowledge of container...Full timeWork experience placementRemote workWorldwideWilson Sonsini Goodrich & Rosati
San Francisco, CA4 days ago- ...business will answer their phone with voice AI. We are making sure that when they build... ...high-pressure environment to work in our engineering. This role is critical to the success of... .... Qualifications: 5+ years as a DevOps engineer Experience writing async web...Work at office
Phonely
San Francisco, CA4 days ago $153k - $202k
...Join to apply for the DevOps Engineer role at Merge Merge is the leading provider of agentic tools and customer-facing integrations for frontier... ...with a single API, and Merge Agent Handler, which empowers AI agents with secure access to thousands of third-party tools....Full timeWork at officeHome officeMerge LLC
San Francisco, CA2 days ago- ...OpenAI is seeking an experienced Engineer to define and build CI/CD systems that qualify frontier models on our custom accelerators. You will establish scalable CI/CD, GitOps, and monorepo practices, and partner with cross‑functional teams to shorten feedback loops without...
$155.4k - $244.2k
...and multi-quarter roadmap for cloud infrastructure, aligned with engineering and business goalsLead architecture and design for cross-team... ...compliance, and policy as code into the platform by defaultApply AI and agent-based tooling to automate operations and improve...Work experience placementWorldwideZoomInfo
San Francisco, CA7 hours ago- ...OpenAI’s Hardware organization seeks an engineer to define and build scalable CI/CD and GitOps infrastructure for frontier models running on OpenAI’s custom accelerators. You will work across model workloads, accelerator software, systems infrastructure, and developer...
- ...is the universal data layer for AI that helps 300+ AI teams train and run models on the right data. Our platform... ...indexes, curates, annotates, and evaluates data across the full AI lifecycle... ...The role We're looking for a DevOps Engineer with 4–5 years of experience to...Work at officeFlexible hours
Encord
San Francisco, CA2 days ago $150k - $200k
...Senior DevOps EngineerYou'll have direct responsibility for critical infrastructure decisions that affect the entire engineering organization. This isn't a role where you're just closing tickets... ...scale how work gets done. Our Workflow AI platform automatically captures and...Work at officeImmediate startRemote workHome officeFlexible hours3 days per weekScribe
San Francisco, CA1 day ago- ...investors, we’re building the category-defining AI workflow automation platform that... ...The Role Plenful is hiring a Senior DevOps Engineer to join our engineering team and help build... ...and New York. R&D roles follow a hybrid model, with two days per week in our San...Full timeWork at officeLocal areaRemote workFlexible hours2 days per week
RXinsider
San Francisco, CA16 hours ago - ...on a mission to create the world’s first AI-powered Personal & Entrepreneurial Resource... ...the way. The Role As a Release Engineer at Air Apps, you will be responsible for... ...involve collaborating with development, QA, DevOps, and product teams to streamline and...Temporary workWorldwide
Air Apps
San Francisco, CA4 days ago $136.44k - $265.11k
...are rebuilding biotech for the AI era.When a breakthrough is... ...data, and run AI agents and models directly in their workflows.... ...here.You’ll build the datasets, evaluations, and systems that help close... ...the intersection of software engineering, biology, and frontier AI: finding...Work at officeLocal areaMonday to FridayShift workBenchling
San Francisco, CA2 days ago$160k - $250k
...creating cutting-edge AI/ML products that solve... ...for a seasoned DevSecOps engineer with significant experience... ...SLA Service cost modeling and ongoing cost... ...Configurations, etc. ~ Solid DevOps experience, preferably... ...applicable laws. We evaluate candidates holistically...Work at officeLocal areaFlexible hoursEllipsis Health
San Francisco, CA2 days ago- ...About Ema Ema builds AI Employees for HR, IT and Finance. Our AI Employees take... ...Coinbase, Flipkart, and Okta, our team includes engineers from premier tech companies and graduates... ...Qualifications Required 4+ years DevOps, SRE, or platform engineering experience...
$172.5k - $260.1k
...SalesforceSalesforce is the #1 AI CRM, where humans with agents... ...of Salesforce.The MeshMesh Engineering team is building the infrastructure... ...networking platform. As a DevOps & Infra Engineer, you will... ...help our recruiters assess and evaluate candidates’ resumes and...Full timeSalesforce
San Francisco, CA1 day ago$246k - $271k
...Cloud DevOps EngineerYou'll build the cloud infrastructure that turns... ...and search products. We need engineers who can make that foundation —... ...way to turn the web into data AI agents can use. One API call... ...infrastructure experience (GPU workloads, model deployment pipelines) is a...Full timeTemporary workFor contractorsRemote workVisa sponsorshipFlexible hoursFirecrawl
San Francisco, CA3 days ago- Obsidian is hiring expert Evaluators in Healthcare operations for a remote, hourly engagement. You will review AI-generated work products for accuracy, rigor, and domain quality, leveraging your extensive subject-matter expertise in the field. The ideal candidate has over...Remote jobHourly payWork at office
$255k - $300k
...AreNotion is the collaborative AI workspace where teams and... ...ChatGPT launched. The job of the Model Capabilities team is to keep us... ...available for our users and our engineers.Make inference reliable:... ...better" means, build or use the evaluation to test it, and are willing to...Local areaNotion Labs
San Francisco, CA7 hours ago$246.8k - $333.9k
...this role, you will lead the AWS account team for a frontier AI model lab and one of Amazon's largest strategic technology partnerships... ...AI/ML)Preferred qualification - Bachelor's degree or above in engineering, management, or technology, or Bachelor's degree in computer...Contract workLocal areaWorldwideFlexible hoursAmazonWebServices
San Francisco, CA4 days ago$166k - $225k
...building and running the world's best data and AI infrastructure platform so our customers... ...to improve their business. Databricks’ Model Serving product provides enterprises with... ...strong SLAs and cost efficiency.As a Senior Engineer, you’ll play a critical role in shaping...Local areaWorldwideDataBricks
San Francisco, CA1 day ago- ...Ema builds AI Employees for HR, IT and Finance. You will build and manage the infrastructure powering our AI platform, designing cloud systems, automating deployments, and ensuring scalable, reliable operations for millions of enterprises. Own Kubernetes clusters,...
- Figma is seeking an AI Model Designer to bridge AI research and design principles across products. You will help teach models to be better copilots for designers and oversee evaluation quality. The role influences design capabilities, requiring collaboration with researchers...Remote jobFull time
$192k - $260k
...building and running the world's best data and AI infrastructure platform so our customers... ...to improve their business. Foundation Model Serving is the API Product for hosting... ...experience is necessary. We’re looking for engineers who have owned high scale operational...Local areaWorldwideDataBricks
San Francisco, CA1 day ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator. Be the first to apply!
Related searches
- devops engineer San Francisco, CA
- senior devops engineer remote San Francisco, CA
- big data devops engineer San Francisco, CA
- devops aws developer (remote) San Francisco, CA
- devops engineer sre San Francisco, CA
- devops engineer full time San Francisco, CA
- senior devops cloud engineer San Francisco, CA
- senior devops engineer San Francisco, CA
- ai evaluator San Francisco, CA
- education evaluator San Francisco, CA