DevOps Engineer - AI Model Evaluator
$85 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
- Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
- Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , observability , and infrastructure automation .
- Identify bugs, edge cases, reliability issues, and failure modes in model outputs.
- Compare outputs from multiple frontier models to assess their strengths and weaknesses.
- Apply professional engineering judgment to realistic infrastructure engineering scenarios.
Qualifications
Must-Have
- 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
- Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling.
- Regular use of AI coding agents such as Cursor, Claude Code, Codex, Windsurf, Gemini CLI, or similar tools.
- Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
- Experience supporting production-scale systems.
Compensation & Legal
- $400 per accepted task
- Compensation tied to accepted work.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on us.fitly.work
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator in New York, NY vacancy
- Dorado is seeking an experienced Investment Banking SME to support the development, evaluation, and improvement of advanced AI models in finance. You will assess AI-generated analyses for accuracy, reasoned judgments, and data integrity, guiding model refinements and prompts...SuggestedRemote job
- Obsidian is seeking Insurance domain SMEs to join a cutting-edge AI training program. You will evaluate AI model outputs against real underwriting practice and rubrics, guiding research and engineering teams to close knowledge gaps in underwriting, claims, and risk assessment...Suggested
$400 per month
...Mercor is partnering with a leading AI research lab to support a Frontier... ...project. Contributors help evaluate and improve frontier AI coding models through structured technical assessments... ...work focuses on realistic data engineering workflows and model evaluation. Spots...Suggested$20 per hour
Feedinkoo is looking for a Web Developer/Designer to enhance AI models by evaluating design work, including interfaces and user experiences. This role involves reviewing AI‑generated visuals and providing feedback to improve users' experience with AI tools. Working remotely...SuggestedRemote job$80 per hour
...and technical talent with leading AI research labs. Headquartered in... ...Jack Dorsey . Position: Data Engineer (Coding Agent Experience) Type:... ...AI coding agents to complete and evaluate complex data engineering tasks. Review model-generated implementations...SuggestedContract workSummer workRemote work- Mercor is seeking a Generalist who can operate in English and Punjabi. This contract, remote position focuses on evaluating AI outputs and supporting model evaluation tasks. You will conduct fact-checking, assess reasoning, clarity, tone and completeness, and provide actionable...Remote jobContract work
$20 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...external tools. Generate high-quality human evaluation data by identifying response strengths, areas... ..., and completeness of responses. Ensure model responses align with expected...Remote jobContract workPart timeSummer work- ...We’re seeking a future team member for the role of SVP - Model Risk Management AI, Wealth and Investment to join our Model Risk team. This role... ...expectations. Advise senior business, risk and Engineering stakeholders on model risk issues and emerging quantitative...Worldwide
BNY
New York, NY13 hours ago - MODEL RISK MANAGEMENT (MRM)The Model Risk Management (MRM) group is a multidisciplinary... ...Goldman Sachs is seeking a highly motivated AI Model Risk Vice President to join our... ...Computer Science, Mathematics, Physics, Engineering, or a closely related quantitative field....Work experience placement
- ...platforms with our developer community, driving adoption of modern DevOps tooling and AI-powered workflows like GitHub Copilot. You'll blend... ...skills to create engaging content, host trainings, and empower engineers to build faster and smarter.At EPAM, you'll work on cutting...
$11.5 per hour
...an Online Task Contributor. In this role, you will evaluate and provide feedback on content to enhance search engine results and quality. No prior experience is needed... ...completion, with a supportive community of contributors involved in AI advancements. #J-18808-LjbffrHourly payPart timeRemote workUniversity of Delaware
New York, NY3 days ago- ...Obsidian is hiring expert Evaluators in real estate, hospitality, and events to review AI-generated work products for accuracy and quality. This is a remote position requiring strong expertise in the relevant domains. Ideal candidates should have over 5 years of professional...Work at officeRemote work
Obsidian
New York, NY13 hours ago $228.7k - $343.1k
...at enormous scale, and one bad model can mean millions in credit... ...you apply to models applies to AI. We build the tooling that lets... ...validate at scale, so you critically evaluate what it produces and own the... .... Solid software and data engineering: production-quality Python,...Remote jobFull timeLocal areaShift workBlock
New York, NY13 hours ago$175k - $215k
...across 15+ U.S. states. The mission of the Waymo AI Foundations team is to develop machine learning... ..., learning from demonstration, generative modeling, Bayesian inference, hierarchical learning, and robust evaluation. In this hybrid role, you will report to a Senior...Full timeRemote workWaymo
New York, NY13 hours ago- ...We are seeking a self-driven Junior DevOps Engineer to contribute to the development, deployment, and maintenance of data and AI products that support positive social impact around the world. You will support the maintenance of the cloud infrastructure that powers our...
Patrick J. McGovern Foundation - Remote, United States
New York, NY4 days ago $140k - $260k
...activity across blockchains?Elliptic is looking for a Senior DevOps Engineer to join our DevOps team. This is a hands-on technical role, working... ...crypto activity. Current major projects include delivering AI Gateway and supporting LangChain/LangGraph infrastructure for...Remote workHome office- ...City-based organization is building an agentic AI platform transforming financial services. As a Cloud Infrastructure Engineer, you will design and manage scalable, secure... ...to cutting-edge AI workloads, modern DevOps practices, and real-world impact on enterprise...Full timeRemote work
- ...A leading AI language services provider is looking for detail-oriented linguists with native speaker fluency in Indonesian for remote freelance support in AI-related projects focusing on prompt evaluation, video understanding, and text review. The ideal candidates will...FreelanceRemote work
LILT (Production)
New York, NY4 days ago $180k - $225k
As a Software Engineer on the ML Infrastructure team, you will design... ...engineers to integrate and optimize models for production and research... ...to ensure a fair and thorough evaluation of all applicants.About Us:At... ...is to develop reliable AI systems for the world's most important...Full time$14.5 per hour
...Join to apply for the AI Web Search Evaluator role at Welo Data Welo Data works with technology... ...and scalable to supercharge their AI models. As a Welocalize brand, Welo Data leverages... ...in improving the quality of search engine results, ensuring users find the most...Hourly payPart timeImmediate startRemote workWork from home10 hours per weekFlexible hoursWelo Data
New York, NY1 day ago$170k - $190k
As a Senior DevOps Engineer, you will be a critical member of our infrastructure team based in New York, providing earlier-hours coverage and... ...Google Cloud Platform (GCP) infrastructure as we expand our AI capabilities.This role requires 3 days in-office (Tuesday, Wednesday...Work at officeRemote workFlexible hours$180k - $230k
SVP, MLOps / DevOps Engineer - Full Time - Hybrid We’re partnering with our client, a fast-growing... ...scale the infrastructure behind their AI and machine learning platforms. This is... ...engineering, data, and cloud; helping bring AI/ML models into production in a secure, scalable,...Full timeRemote work- ...DevOps Engineer - US is a remote engineering review track for evaluating production code, debugging traces, and developer-facing AI outputs against real-world correctness standards. Reviewers reproduce... ..., write the unit test the model should have written, and explain...Hourly payFor contractorsRemote work10 hours per week
AuraOne
New York, NY5 days ago - ...AWS DevOps Engineer (Container Deployment Specialist) (Unpaid/Remote) job at Cyber Risk Research Institute. New York, NY. Volunteer Opportunity... ...platform that supports large-scale analysis of cybersecurity, AI, and technology governance materials for research, policy, and...Remote workFlexible hours
Fairweather
New York, NY4 days ago - ...DevOps EngineerWe're looking for a DevOps Engineer to build and operate the infrastructure that keeps Themis secure, reliable, and fast as we scale. In this role... ...at a startup or high-growth companyFamiliarity with AI and automation tooling for operationsWhat We ValueThe...Remote workFlexible hoursShift work
Mercury Systems, Trusted Mission Solutions
New York, NY5 days ago $96.3k - $260.1k
...DetailsAbout SalesforceSalesforce is the #1 AI CRM, where humans with agents drive... ...CAC/PIV.We are looking for an experienced DevOps engineer to join a team of highly skilled Systems... ...accurately and reliably.Critically evaluate code (Human or AI-generated) for correctness...Full timeWork experience placementLocal area- ...Senior DevOps EngineerWe're seeking a Senior DevOps Engineer / Site Reliability Engineer to own and evolve our infrastructure... ...workloads on ECS Fargate and evaluate migration paths to KubernetesEstablish... ...the foundation layer of an AI-native productHigh ownership + direct...
Avantos
New York, NY5 days ago $180k - $230k
...creative, and intellectually curious MLOps/DevOps Engineers with deep expertise in machine learning... ...‑grade MLOps platforms, including model lifecycle automation, observability, and... ...monitoring).Enable production workloads for AI/ML and Generative AI systems, including...Full timeWork at officeRemote work$65 - $80 per hour
...Senior DevOps Engineer (GCP) Remote, US, (Eastern Time Preferred) WE WILL NOT BE USING AGENCIES ON THIS ROLE This is an opportunity to play a key role in building a next-generation AI data platform that will support large-scale data processing and enterprise...Hourly payRemote workHarnham
New York, NY3 days ago$160k - $200k
...ownership infrastructure role at a fast-growing AI-native B2B SaaS company serving... ...workloads — working closely with backend and ML engineers to keep things secure, scalable, and... ...across the team. Mentor team members on DevOps best practices and help guide broader...Clera
New York, NY13 hours ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator. Be the first to apply!
Related searches
- devops engineer contract New York, NY
- senior devops cloud engineer New York, NY
- senior devops engineer New York, NY
- devops engineer sre New York, NY
- devops cloud engineer New York, NY
- devops engineer azure New York, NY
- devops engineer New York, NY
- devops engineer remote New York, NY
- senior devops engineer remote New York, NY
- devops aws developer (remote) New York, NY