DevOps Engineer - AI Model Evaluator
$85 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
- Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
- Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , observability , and infrastructure automation .
- Identify bugs, edge cases, reliability issues, and failure modes in model outputs.
- Compare outputs from multiple frontier models to assess their strengths and weaknesses.
- Apply professional engineering judgment to realistic infrastructure engineering scenarios.
Qualifications
Must-Have
- 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
- Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling.
- Regular use of AI coding agents such as Cursor, Claude Code, Codex, Windsurf, Gemini CLI, or similar tools.
- Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
- Experience supporting production-scale systems.
Compensation & Legal
- $400 per accepted task
- Compensation tied to accepted work.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on ziprecruiter.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 15 days ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator in New York, NY vacancy
$85 per hour
...technical talent with leading AI research labs. Headquartered... ...Dorsey . Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)... ...coding agents to complete and evaluate complex infrastructure engineering tasks. Review model-generated implementations involving...SuggestedContract workSummer workRemote work- Mercor is seeking board-certified critical care physicians (MDs/DOs) to evaluate health AI models in partnership with a leading AI lab. You will read de-identified patient charts and judge AI answers to clinical questions in critical care. Responsibilities include evaluating...SuggestedRemote jobHourly payPart timeImmediate start
- Mercor is seeking board-certified psychiatrists (MDs and DOs) to evaluate health AI models remotely in the United States. You will assess AI-generated psychiatric answers for accuracy and safety, reading de-identified patient charts. We are looking for active US licenses...SuggestedRemote jobPart timeImmediate startFlexible hours
- Mercor is seeking experienced MDs or DOs to evaluate health AI models remotely in the United States. You will review outputs across diagnoses, clinical reasoning and care workflows, and provide structured feedback to ensure medical accuracy and safety. The role requires...SuggestedRemote jobPart timeFor contractors
- AI Trainer Jobs is seeking a remote contractor to evaluate people ops / recruiting prompts and responses against a evolving quality rubric. You will compare model outputs, label edge cases, and provide structured feedback for retraining. Responsibilities include evaluating...SuggestedRemote jobFor contractors
- Table Image Reasoning Model Evaluator is a remote evaluation track for reviewing table image reasoning model evaluation prompts and responses... ...the modeling team can use to retrain. Why this role matters AI data reviewers help turn table image reasoning model evaluation...Hourly payFor contractorsRemote work10 hours per week
- ...dynamic resource: Description: The Lead DevOps Engineer is a deeply technical role responsible... ..., you will work cross-functionally with AI and ML development teams to design,... ...Langfuse, LangGraph/LangChain, MCP servers, model invocation cost tracking)....Full timeRemote work
ePlus inc.
New York, NY2 days ago $150k - $185k
...Site Reliability Engineer — Cloud / DevOps NYC | Hybrid 4 Days Onsite | Mid–Senior | $150,000–$185,000 We're a fast-growing fintech platform transforming... ...position. No C2C, no consultants, no visa sponsorship. AI workload infrastructure is on our roadmap. If that interests...Full timeImmediate startVisa sponsorshipLaStellar Group
New York, NY2 days ago- We’re seeking a future team member for the role of SVP - Model Risk Management AI, Wealth and Investment to join our Model Risk team. This role... ...regulatory expectations.Advise senior business, risk and Engineering stakeholders on model risk issues and emerging...WorldwideFlexible hours
The Bank of New York Mellon
New York, NY2 days ago - ...Job Description Job Title: Language Model Evaluator Job Summary We are seeking a Language Model... ...Language Models (LLMs) and Generative AI applications. The ideal candidate will evaluate... ...while collaborating with AI Engineers, Data Scientists, Prompt Engineers, and...Full timeRemote work
Ova Technologies
New York, NY2 days ago $140k - $200k
...industries. Our capabilities span Data & AI, Cybersecurity, Risk & Compliance, Digital... ...We combine deep industry expertise with engineering excellence to help clients modernize platforms... ...Engineers to support a high-profile Model Risk transformation program at a leading...Temporary workRemote workWorldwideBIP US
New York, NY4 days ago- ...DevOps Engineer We're building the infrastructure layer for agentic web interaction at scale. Our API is designed from the ground up to... ...Retrieval-Augmented Generation (RAG) and real-time reasoning in AI systems. By connecting LLMs to high-quality, trustworthy web content...
Tavily Inc
New York, NY16 hours ago $130k - $190k
...Senior Release Engineer, Capital Markets Company Overview Business Integration Partners... ...Payments verticals, supplemented with Data & AI, Cybersecurity, Risk & Compliance, Change... ...preferred. ~ SAFe Agilist, SAFe DevOps Practitioner, or Release Train Engineer certification...Full timeTemporary workRemote workWorldwideBIP US
New York, NY2 days ago$131.75k - $178.25k
...professional achievement and career growth. The Senior Infrastructure DevOps Engineer, under the direction of the Senior Manager of Infrastructure,... ...Apps.Experience with managing O365 services.Experience with AI technologies such as Copilot or Claude.Knowledge of container...Full timeWork experience placementRemote workWorldwideWilson Sonsini Goodrich & Rosati
New York, NY16 hours ago- MODEL RISK MANAGEMENT (MRM)The Model Risk Management (MRM) group is a multidisciplinary... ...Goldman Sachs is seeking a highly motivated AI Model Risk Vice President to join our... ...Computer Science, Mathematics, Physics, Engineering, or a closely related quantitative field....Work experience placement
Goldman Sachs
New York, NY1 day ago $100k - $140k
...to the Board and protecting the Firm from exposure to losses as a result of credit, market, liquidity, operational, model and other risks.Associate - AI Governance & ReportingPosition SummaryThe Associate will support the delivery of AI-enabled solutions across Model Risk...Temporary workMorgan Stanley
New York, NY16 hours ago- ...Senior DevOps Engineer Stack AI is building cutting-edge infrastructure at the intersection of AI and enterprise. Our mission is to enable companies to safely and reliably deploy AI at scale. As we grow, we're looking for a Senior DevOps Engineer who can design, build...Work at officeImmediate startRemote work
Stack AI
New York, NY3 days ago $100k - $160k
...system. Today it is the platform that drives AI adoption across the enterprise.... ...there, WalkMe is the place for you!As a DevOps Engineer at WalkMe, you'll join our Cloud Engineering... ...maintain smooth operations.Participate in evaluating and integrating new technologies to...Full timeWork experience placementLocal areaImmediate startRemote workWalkMe
New York, NY3 days ago$96.3k - $260.1k
...DetailsAbout SalesforceSalesforce is the #1 AI CRM, where humans with agents drive... ...CAC/PIV.We are looking for an experienced DevOps engineer to join a team of highly skilled Systems... ...accurately and reliably.Critically evaluate code (Human or AI-generated) for correctness...Full timeWork experience placementLocal areaSalesforce
New York, NY3 days ago$65 - $80 per hour
...Senior DevOps Engineer (GCP) Remote, US, (Eastern Time Preferred) WE WILL NOT BE USING AGENCIES ON THIS ROLE This is an opportunity to play a key role in building a next-generation AI data platform that will support large-scale data processing and enterprise...Hourly payRemote workHarnham
New York, NY16 hours ago- ...il territorio nazionale, ricerca per importante cliente un/una DevOps Engineer Sede: Remoto Requisiti Almeno 3 anni di esperienza nel ruolo... ...finalità di selezioni del personale per esigenze attuali e future ai sensi del DL 196 del 20/06/2003 e dell Art. 13 GDPR 679/16....Remote work
- ...Developer Infrastructure Engineer Build the developer infrastructure that powers one of... ...productivity, systems infrastructure, and DevOps practices. Maintain and improve build... ...C++ and/or Rust. Experience integrating AI-assisted development tools into...Work at officeRemote work
Catch Talent
New York, NY3 days ago - ...Medical Translation AI Evaluator is a remote clinical-review track for evaluating AI outputs that touch clinical review. Reviewers grade... ...issues; and document the corrected clinical reasoning so the modeling team can close the gap. Why this role matters Clinical review...Hourly payFor contractorsRemote work10 hours per week
AuraOne, Inc.
New York, NY16 hours ago - ...MERIT Beauty is seeking Turkish-speaking annotators for a contract role evaluating AI-generated content. You will review content for coherence, consistency, and alignment with real-world expectations in Turkish. The ideal candidates are native or fluent Turkish speakers...Contract workTemporary workImmediate startRemote work
MERIT Beauty
New York, NY4 days ago - ...Senior DevOps Engineer We are looking for a Senior DevOps Engineer to join our DevOps team at... ...infrastructure underpinning a healthcare AI platform serving patients and enterprise... ...care engine. Our deeply integrated model modernizes the primary care loop by using...Work at officeLocal areaShift work
K Health
New York, NY2 days ago $68.21k
...experienced and forward-thinking Senior DevOps Engineer to drive the modernization, scalability,... ...validation, and deployment of cutting-edge AI models, Model Context Protocol (MCP) servers and Proofs of Concept (POCs) to evaluate technical feasibility and accelerate business...Permanent employmentFull timeWork at officeShift workNight shiftWeekend workAfternoon shiftCity of New York
New York, NY2 days ago$130k - $196.5k
...Platform and Identity Engineering LiveRamp is shaping... ...consumers embrace new AI-driven experiences, the... ...LiveRamp.com. As a Senior DevOps Engineer, you will join... ..., and Large Language Models (LLMs) into the... ...ready infrastructure. Evaluate, introduce, and standardize...Full timeWorldwideLiveRamp
New York, NY3 days ago- Agency seeks an experienced voice actor to help shape AI voice models through training data. You will record scripted material, evaluate AI outputs for pronunciation, tone, and pacing, and provide feedback to refine prompts and voice design guidelines. Prior work in voice...Remote jobContract workFreelance
- ...and a brand they'll be proud to be part of. Confido is the AI operating system that runs cash application, deductions, trade... ...is down or slow, a customer's cash gets stuck. We're hiring a DevOps Engineer to own the infrastructure and tooling that keeps the platform...Work at officeLocal areaRelocation packageWeekend work
Confido
New York, NY3 days ago $40 - $60 per hour
...Compensation: $40-$60/hour Domain: Linux | Infrastructure | DevOps | Kubernetes | Systems Engineering | Terminal Bench Role Overview We are looking for... ...systemd, LVM, security hardening, distributed systems, or AI coding agents is preferred. #J-18808-Ljbffr CodefeastContract workImmediate startRemote work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator. Be the first to apply!
Related searches
- senior devops engineer remote New York, NY
- senior devops cloud engineer New York, NY
- devops engineer full time New York, NY
- devops engineer sre New York, NY
- big data devops engineer New York, NY
- devops engineer remote New York, NY
- devops cloud engineer New York, NY
- devops aws developer (remote) New York, NY
- devops engineer contract New York, NY
- devops engineer New York, NY