DevOps Engineer - AI Model Evaluator

$85 per hour

Mercor

New York, NY

Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , observability , and infrastructure automation .
Identify bugs, edge cases, reliability issues, and failure modes.
Compare outputs from multiple frontier models and assess their strengths and weaknesses.
Apply professional engineering judgment to realistic infrastructure engineering scenarios.
Work independently and asynchronously to meet sprint-based project deadlines.
Qualifications
Must-Have
2+ years of professional DevOps , SRE , or Cloud Engineering experience.
Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling .
Regular use of AI coding agents such as Cursor, Claude Code, Codex, Windsurf, Gemini CLI, or similar tools.
Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
Experience supporting production-scale systems.
Compensation & Legal
$400 per accepted task .
Compensation is tied to accepted work.
Application Process (Takes 20–30 mins to complete)
Upload resume
AI interview based on your resume
Submit form
Resources & Support
For details about the interview process and platform information, please check:
For any help or support, reach out to: View email address on us.fitly.work
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

Apply

Vacancy posted 5 days ago

Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator in New York, NY vacancy

Data Engineer - AI Model Evaluator
$80 per hour
...and technical talent with leading AI research labs. Headquartered in... ...Jack Dorsey . Position: Data Engineer (Coding Agent Experience) Type:... ...AI coding agents to complete and evaluate complex data engineering tasks. Review model-generated implementations...
Suggested
Contract work
Summer work
Remote work
Mercor
New York, NY
16 days ago
Senior DevOps Platform Engineer
$106.9k - $147k
...communityHumana is looking for a Senior DevOps Platform Engineer to design, implement, and support... ...solutions using PowerShell, Bash, Python, and AI-assisted engineering tools.Ensure HIPAA... ...with DORA metrics and DevOps maturity models.Experience integrating automated...
Suggested
Full time
Temporary work
Apprenticeship
Work at office
Remote work
Home office
Humana
New York, NY
3 days ago
Swahili Evaluation AI Evaluator
...About the role Swahili Evaluation AI Evaluator is a remote evaluation track for reviewing swahili generalist evaluation prompts and responses... ...edge cases, and write the kind of structured feedback the modeling team can use to retrain. Category: Frontier Model Evaluation...
Suggested
Hourly pay
For contractors
Remote work
10 hours per week
AI Trainer Jobs
New York, NY
1 day ago
DevOps Enablement Engineer & AI Adoption Specialist
...platforms with our developer community, driving adoption of modern DevOps tooling and AI-powered workflows like GitHub Copilot. You'll blend... ...skills to create engaging content, host trainings, and empower engineers to build faster and smarter.At EPAM, you'll work on cutting...
Suggested
EPAM Systems
New York, NY
16 hours ago
Remote Turkish Language Evaluator - AI Content Quality
...MERIT Beauty is seeking Turkish-speaking annotators for a contract role evaluating AI-generated content. You will review content for coherence, consistency, and alignment with real-world expectations in Turkish. The ideal candidates are native or fluent Turkish speakers...
Suggested
Contract work
Temporary work
Immediate start
Remote work
MERIT Beauty
New York, NY
1 day ago
Associate - AI Governance & Reporting - Model Risk
$100k - $140k
...to the Board and protecting the Firm from exposure to losses as a result of credit, market, liquidity, operational, model and other risks.Associate - AI Governance & ReportingPosition SummaryThe Associate will support the delivery of AI-enabled solutions across Model Risk...
Temporary work
Morgan Stanley
New York, NY
1 day ago
Remote AI Quality Evaluator - Real Estate & Hospitality
...Obsidian is hiring expert Evaluators in real estate, hospitality, and events to review AI-generated work products for accuracy and quality. This is a remote position requiring strong expertise in the relevant domains. Ideal candidates should have over 5 years of professional...
Work at office
Remote work
Obsidian
New York, NY
3 days ago
AI Web Search Evaluator
$14.5 per hour
...Join to apply for the AI Web Search Evaluator role at Welo Data Welo Data works with technology... ...and scalable to supercharge their AI models. As a Welocalize brand, Welo Data leverages... ...in improving the quality of search engine results, ensuring users find the most...
Hourly pay
Part time
Immediate start
Remote work
Work from home
10 hours per week
Flexible hours
Welo Data
New York, NY
4 days ago
Senior DevOps Engineer
$170k - $190k
As a Senior DevOps Engineer, you will be a critical member of our infrastructure team based in New York, providing earlier-hours coverage and... ...Google Cloud Platform (GCP) infrastructure as we expand our AI capabilities.This role requires 3 days in-office (Tuesday, Wednesday...
Work at office
Remote work
Flexible hours
Dealpath
New York, NY
4 days ago
Senior Infrastructure DevOps Engineer
$131.75k - $178.25k
...professional achievement and career growth. The Senior Infrastructure DevOps Engineer, under the direction of the Senior Manager of Infrastructure,... ...Apps.Experience with managing O365 services.Experience with AI technologies such as Copilot or Claude.Knowledge of container...
Full time
Work experience placement
Remote work
Worldwide
Wilson Sonsini Goodrich & Rosati
New York, NY
16 hours ago
DevOps Engineer III
$116.2k - $229.1k
Position Summary AI & Engineering/EaaS - DevOps Engineer IIIBuild and scale modern DevOps capabilities that help clients accelerate delivery... ...engineering processes and define target-state DevOps operating models, tooling, and workflowsBuilding and maintaining cloud...
Local area
Deloitte
New York, NY
2 days ago
DevOps Engineer, DevEx
...-world safety is impacted. As a Platform Engineer, you will take ownership of critical platform... ...of experience in Platform Engineering, DevOps, and/or Developer Experience ~... ...optimization ~ Experience integrating AI/LLM capabilities into developer workflows...
Local area
Immediate start
Remote work
Work visa
Mark43
New York, NY
a month ago
GCP Devops Engineer
...are seeking a hands-on Cloud Engineer GCP Cloud to own the end-to-end... ...foundation powering our Agentic AI Solution Platform.Key... ...Active/Active-Passive deployment models.Security Identity ManagementDesign... ...modules and automated IaC pipelines.DevOps & Scripting: Proficient in...
Work experience placement
Virtusa
New York, NY
3 days ago
Senior DevOps Engineer
$200k - $250k
...: We're looking for a senior level DevOps Engineer to join a small, high-impact team that builds... ..., and help build a sustainable support model as the platform scales. Requirements:... ..., storage systems. Thoughtful use of AI coding assistants and interest in AI-...
Worldwide
Flexible hours
DV Trading
New York, NY
a month ago
Release Train Engineer
...and apply Agile and flow metrics Use AI-enabled insights for forecasting, capacity... ...experience, including significant Release Train Engineer experience across product teams ~5+... ...improvement ~ Experience with Azure DevOps, JIRA, Confluence, Power BI, Microsoft Teams...
Jobtailor
New York, NY
17 hours ago
Remote Legal Expert for AI Model Training
$100 per hour
...Prolific is seeking Legal Subject Matter Experts to join our Expert Network and help train AI models with professional legal insights. You’ll complete a brief test, then join Prolific's Legal Projects as a participant for paid tasks, with up to $100/hour depending on skills...
Remote work
Prolific Academic Ltd
New York, NY
2 days ago
DevSecOps Engineer
$140k - $195k
...minded, automation-first DevSecOps Engineer to embed security into every stage of LockedIn AI’s software development and... ...strong software engineering and DevOps skills with deep security expertise... ...secure coding standards, threat modeling, and security reviews early in the...
Shift work
Lockedinai
New York, NY
17 hours ago
Senior DevOps Engineer — AI-Driven Cloud Infra
...Salesforce is seeking an experienced DevOps engineer to join a team focused on Government Cloud infrastructure. You will ensure maximum availability... ...performance, and uphold security compliance while advancing AI-powered development workflows. You’ll work across teams on...
Jobleads-US
New York, NY
1 day ago
DevOps & AI Adoption Engineer
$65k - $102k
..., Technical Training, or Technical Writing with a focus on DevOps or Platform Engineering Strong grasp of modern DevOps practices, including CI/CD... ...version control systems Practical experience using generative AI tools like GitHub Copilot or ChatGPT for development,...
Full time
Work at office
EPAM Systems
New York, NY
a month ago
Senior DevOps / Infrastructure Engineer (Americas, US time zones)
..., and English is the working language of engineering. Why US time zones Our engineering team... ...the Role We are looking for a Senior DevOps / Infrastructure Engineer to own the infrastructure... ...other latency-sensitive systems; using AI tools in infrastructure workflows; Go,...
Full time
Remote work
Startale Group
New York, NY
5 days ago
Data Analysis Expert - Evaluator
...About the role We are hiring expert Evaluators in Data analysis / quantitative readouts to review and assess AI-generated work products (documents, spreadsheets, and slide decks) for accuracy, rigor, and domain quality. You will apply deep subject-matter expertise...
Hourly pay
Work at office
Remote work
Mercor Inc
New York, NY
2 days ago
Senior DevSecOps Engineer
$170k - $230k
...Cybersecurity Lead, the Senior DevSecOps Engineer will design, deploy, and... ...and extends these controls to AI/LLM development and inference... ...gates, including threat modelling and secure design practices.... ...automation using GitLab and/or Azure DevOps, including automated security...
Work at office
DFO Referrals
New York, NY
1 day ago
Senior DevOps Engineer
$130k - $145k
...Job Description Job Description Job Summary The Senior DevOps Engineer at Delta Dental of New Jersey & Connecticut plays a central role... ...to DDNJ's modernization efforts, including the adoption of AI-assisted engineering tooling. This is a hybrid role, with...
Temporary work
Work at office
Remote work
2 days per week
Delta Dental of New Jersey, Inc.
New York, NY
a month ago
DevOps Enablement Engineer & AI Adoption Specialist
...platforms with our developer community, driving adoption of modern DevOps tooling and AI-powered workflows like GitHub Copilot. You'll blend... ...skills to create engaging content, host trainings, and empower engineers to build faster and smarter. At EPAM, you'll work on...
Work at office
EPAM Systems
New York, NY
a month ago
Release Train Engineer
$170k - $190k
...delivery while embedding innovative AI-enabled practices and... ...including significant Release Train Engineer experience across product... ...management tools such as Azure DevOps, JIRA, Confluence, Power BI, Microsoft... ...tracking market performance, evaluating portfolios and developing...
Live in
Worldwide
Flexible hours
S&P Global
New York, NY
4 days ago
Release Train Engineer
...Release Train Engineer Choosing Capgemini means choosing a company where you will be empowered... .... ~ Experience with Jira, Azure DevOps, Confluence, Rally, or similar tools. ~... ...fueled by its market leading capabilities in AI, generative AI, cloud and data, combined...
Full time
Local area
Capgemini
New York, NY
2 days ago
Principal DevOps Engineer
$184.5k - $246k
...lifecycle, from design to closeout, in a single AI-driven ecosystem. The Medidata Platform... ...and tools relied upon by all internal engineering teams and customer-facing functions at... ...adoption of scalable, compliant (FedRAMP, DR) DevOps practices. Mentor and coach junior and...
Hourly pay
Work at office
Local area
Remote work
Flexible hours
Shift work
Medidata Solutions
New York, NY
3 days ago
ML Engineer - Model Evaluation
$60 - $90 per hour
...connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Summers , and Jack Dorsey . Position: Machine Learning Engineer — Model Evaluation & Experimentation Type: Contract Compensation: $...
Contract work
Summer work
Remote work
Mercor
New York, NY
5 days ago
Health AI Conversation Evaluator - AI Trainer
$20 - $160 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Position: Generalist Annotator — Health AI Conversation Quality Evaluation Type: Contract Compensation: $20–$160/hour Location...
Contract work
Summer work
Remote work
Mercor
New York, NY
5 days ago
Senior Director, Model and AI Review
$195k - $315k
...implementation, and oversight of our enterprise AI governance framework. The ideal candidate... ...to leading and enhancing GEICO’s model and AI governance program, including frameworks... ...controls monitoring and testing to evaluate model performance in line with existing and...
Hourly pay
Work experience placement
Local area
3 days per week
Jobleads-US
New York, NY
4 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator. Be the first to apply!