DevOps Engineer - AI Model Evaluator - AI Trainer
$85 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
- Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
- Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , observability , and infrastructure automation .
- Identify bugs, edge cases, reliability issues, and failure modes.
- Compare outputs from multiple frontier models and assess their strengths and weaknesses.
- Apply professional engineering judgment to realistic infrastructure engineering scenarios.
- Work independently and asynchronously to meet sprint-based project deadlines.
Qualifications
Must-Have
- 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
- Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling .
- Regular use of AI coding agents such as Cursor, Claude Code, Codex, Windsurf, Gemini CLI, or similar tools.
- Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
- Experience supporting production-scale systems.
Compensation & Legal
- $400 per accepted task .
- Compensation is tied to accepted work.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on us.fitly.work
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
#hiringmercorVacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator - AI Trainer in San Francisco, CA vacancy
$400 per month
...partnering with a leading AI research lab to support a... .... Contributors help evaluate and improve frontier AI coding models through structured technical... ...realistic infrastructure engineering workflows and model evaluation... ...2+ years of professional DevOps, SRE, or Cloud...Suggested$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Dorsey . Position: Incident management / reliability / SRE Evaluator Type: Contract Compensation: $80–$120/hour Location...SuggestedContract workSummer workWork at officeRemote work- ...Overview We are looking for Generalists with strong analytical and critical-thinking skills to support AI and data-focused projects. The role involves reviewing, evaluating, and annotating information across different topics while maintaining high accuracy and quality....SuggestedRemote jobFor contractors
$90 per hour
...technical talent with leading AI research labs. Headquartered in... .... Position: Frontend Engineer — Web Replication Preference Rater... ...fidelity in detail, including box model, typography, color, image... ...that merely look correct. Evaluate responsive behavior and semantic...SuggestedContract workSummer workLocal areaRemote work$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...customer research and feedback synthesis Evaluator Type: Contract Compensation: $... ...asynchronously to meet deadlines while improving AI model performance . Utilize Microsoft...SuggestedContract workSummer workWork at officeRemote work$100 - $150 per hour
...technical talent with leading AI research labs. Headquartered in... .... Position: Software Engineer Expert Type: Contract... ...Remote Role Responsibilities Evaluate AI-generated artifacts for usability... ...feedback to improve AI model outputs . Apply deep subject...Contract workSummer workWork at officeRemote work- ...Role Title: College Student, AI Robotics Trainer (No Experience Required) Role Type: Contractor Location: On-site, San Francisco micro... ...train next-generation AI systems. Your work will shape how models learn, reason, and perform through high-quality, real-world...Temporary workFor contractors
micro1
San Francisco, CA15 days ago - ...is building the world’s leading Agentic AI platform to transform enterprise productivity... ..., Flipkart, and Okta, our team includes engineers from premier tech companies and graduates... ...scalingQualificationsRequired4+ years DevOps, SRE, or platform engineering experienceExpert...Remote work
Ema
San Francisco, CA3 days ago $90 per hour
...technical talent with leading AI research labs. Headquartered in... .... Position: Frontend Engineer Type: Contract Compensation... ...fidelity in detail: box model and spacing, typography (family... ..., preference-labeling, model-evaluation, or structured code-review work...Contract workSummer workLocal areaRemote workMercor
San Francisco, CA4 days ago$80 per hour
...and technical talent with leading AI research labs. Headquartered in... ...Jack Dorsey . Position: Data Engineer (Coding Agent Experience) Type:... ...AI coding agents to complete and evaluate complex data engineering tasks. Review model-generated implementations...Contract workSummer workRemote workMercor
San Francisco, CA4 days ago$30 per hour
A leading platform for AI training is seeking advanced Japanese speakers to join as Domain... ...complete various AI training tasks and evaluate AI responses in Japanese. Compensation is... ...a dynamic environment aimed at improving powerful AI models. #J-18808-Ljbffr ProlificRemote jobHourly payWork from homeFlexible hours$150k - $200k
...Get AI-powered advice on this job and more exclusive features. Understanding Recruitment provided pay range This range is provided... ...working with a leading AI company, who are looking for a DevOps Engineer to join their ever-expanding Engineering team who can bridge the...Full timeWork at officeImmediate start2 days per week3 days per weekUnderstanding Recruitment
San Francisco, CA2 days ago$192.2k - $260k
...systems that combine innovative AI, sophisticated control... ...learning and large language models.We leverage advanced robotics... ...source and internal research), evaluate them on representative tasks,... ...reproducible results and solid engineering practices, closing the gap between...Local areaWorldwideFlexible hoursAmazon
San Francisco, CA3 days ago- ...OpenAI is seeking an experienced Engineer to define and build CI/CD systems that qualify frontier models on our custom accelerators. You will establish scalable CI/CD, GitOps, and monorepo practices, and partner with cross‑functional teams to shorten feedback loops without...
$116.2k - $229.1k
Position Summary AI & Engineering/EaaS - DevOps Engineer IIIBuild and scale modern DevOps capabilities that help clients accelerate delivery... ...engineering processes and define target-state DevOps operating models, tooling, and workflowsBuilding and maintaining cloud...Local areaDeloitte
San Francisco, CA1 day ago$183k - $248k
...out of hardware. To do this, we’re building the world's first AI hardware engineer. Founded in 2019, our platform enables anyone to go from... ...and that means everything around it has to just work. As a DevOps Engineer, you’ll work on the full-stack systems that power Flux...Local areaShift workFlux
San Francisco, CAa month ago- ...Tavus, we're building the human layer of AI. Our mission is to make human-AI interaction... ...pioneering research in multimodal AI for modeling human-to-human communication (language,... ...for an experienced Research Scientist/Engineer with a focus on model optimization to join...Full timeRemote workRelocation packageFlexible hours
Tavus
San Francisco, CA4 days ago $70 - $90 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Location: Remote Role Responsibilities Evaluate the quality, correctness, and hardware-... ...tasks for training and evaluating AI models. Assess CUDA→NKI migration fidelity...Contract workSummer workRemote workMercor
San Francisco, CA4 days ago$80 - $90 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...penetration testing , security architecture and engineering , and governance/risk/compliance .... ...cybersecurity methodologies ( threat modeling , incident response playbooks , risk...Contract workSummer workRemote workMercor
San Francisco, CA4 days ago- ...business will answer their phone with voice AI. We are making sure that when they build... ...high-pressure environment to work in our engineering. This role is critical to the success of... .... Qualifications: 5+ years as a DevOps engineer Experience writing async web...Full timeWork at office
Phonely, Inc.
San Francisco, CA4 days ago $160k - $250k
...creating cutting-edge AI/ML products that solve... ...for a seasoned DevSecOps engineer with significant experience... ...SLA Service cost modeling and ongoing cost... ...Configurations, etc. ~ Solid DevOps experience, preferably... ...applicable laws. We evaluate candidates holistically...Work at officeLocal areaFlexible hoursEllipsis Health
San Francisco, CA2 days ago- Obsidian is hiring expert Evaluators in Healthcare operations for a remote, hourly engagement. You will review AI-generated work products for accuracy, rigor, and domain quality, leveraging your extensive subject-matter expertise in the field. The ideal candidate has over...Remote jobHourly payWork at office
$40 per hour
Overview AI Trainer - Advanced SQL Developers About Prolific Prolific is not just another player in the AI space - we are... ...role We’re looking for SQL Developers to help train and evaluate cutting-edge AI models. If you have the necessary experience, we’ll send you an...Remote jobSelf employmentWork from homeFlexible hours$153k - $202k
Join to apply for the DevOps Engineer role at Merge Merge is the leading provider of agentic tools and customer-facing integrations for frontier... ...with a single API, and Merge Agent Handler, which empowers AI agents with secure access to thousands of third-party tools....Full timeWork at officeHome office$85 - $120 per hour
...elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our... ...Commitment: 40 hours/week Role Responsibilities Evaluate the quality of legal knowledge work tasks and AI model outputs . Identify missing behaviors and thin...Hourly payWeekly payContract workFor contractorsSummer workMercor
San Francisco, CA4 days ago$140k - $200k
...costs requires continuous effort and time from DevOps teams. Pump is building the fastest way to save ~60% on cloud spend. Our AI-powered platform not only fully automates... ...run on them, and collaborating with other engineers, designers, and founders. A great...Full timeWork at office$65 - $75 per unit
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...chemical safety , labeled as benign, dual-use, or adversarial. Evaluate model responses against a defined policy standard and judge...Hourly payWeekly payContract workFor contractorsSummer workRemote workMercor
San Francisco, CA4 days ago- ...artificial intelligence company based in San Francisco, California. The Role: As a Research Engineer - Model Architectures , you will be a core contributor to Zyphra’s AI Architecture Research Team. This will involve designing and rigorously testing novel model...Work at officeRelocation package
Zyphra
San Francisco, CAa month ago - ...Associate Vice President, Model & AI Governance About the Company Influential organization in the health & wellness industry Industry Insurance Type Public Company Founded 1964 Employees 10,001+ Categories Business Services Insurance...
- ...s or Bachelor's program with a degree an engineering discipline.Who you'll work withDriving lasting... ...infrastructure that powers cutting-edge AI applications leveraging LLMs, retrieval... ...(Git) and foundational concepts in DevOps and MLOps/LLMOps and CI/CD principlesExceptional...ApprenticeshipInternshipWork at officeLocal areaEasy work
McKinsey & Company
San Francisco, CA3 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator - AI Trainer. Be the first to apply!
Related searches
- devops aws developer (remote) San Francisco, CA
- big data devops engineer San Francisco, CA
- devops engineer San Francisco, CA
- senior devops engineer remote San Francisco, CA
- devops engineer full time San Francisco, CA
- senior devops cloud engineer San Francisco, CA
- ai evaluator San Francisco, CA
- quality evaluator San Francisco, CA
- social media evaluator San Francisco, CA
- education evaluator San Francisco, CA