DevOps Engineer - AI Model Evaluator
$85 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
- Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
- Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , observability , and infrastructure automation .
- Identify bugs, edge cases, reliability issues, and failure modes in model outputs.
- Compare outputs from multiple frontier models to assess their strengths and weaknesses.
- Apply professional engineering judgment to realistic infrastructure engineering scenarios.
Qualifications
Must-Have
- 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
- Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling.
- Regular use of AI coding agents such as Cursor, Claude Code, Codex, Windsurf, Gemini CLI, or similar tools.
- Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
- Experience supporting production-scale systems.
Compensation & Legal
- $400 per accepted task
- Compensation tied to accepted work.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on ziprecruiter.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 25 days ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator in New York, NY vacancy
$85 per hour
...technical talent with leading AI research labs. Headquartered... ...Dorsey . Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)... ...coding agents to complete and evaluate complex infrastructure engineering tasks. Review model-generated implementations involving...SuggestedContract workSummer workRemote work- ...Management & Marketing to remotely review evaluation prompts and responses against the company... ..., and provide structured feedback the modeling team can use to retrain. As a contractor,... ...detail, clear written reasoning, and reliable async #J-18808-Ljbffr AI Trainer JobsSuggestedRemote jobFor contractors
$80 per hour
...and technical talent with leading AI research labs. Headquartered in... ...Jack Dorsey . Position: Data Engineer (Coding Agent Experience) Type:... ...AI coding agents to complete and evaluate complex data engineering tasks. Review model-generated implementations...SuggestedContract workSummer workRemote workMercor
New York, NY9 days ago- Prolific Academic Ltd is seeking Licensed Pharmacists to assist in AI model training and evaluation from a home office. Successful candidates join as Domain Experts and will be paid to train and evaluate models, with tests to assess suitability. Experts review AI-driven...SuggestedRemote workHome officeFlexible hours
$80 - $150 per hour
Prolific Academic Ltd is recruiting Medical Doctors to help train and evaluate AI models. You’ll complete a quick test to assess suitability and, if successful, join as a Domain Expert, paid to work on AI tasks. Researchers pay $80-$150 per hour per completed task, with...SuggestedHourly paySelf employmentWork from home$50 per hour
Combinatorics Model Evaluator is a remote review track for evaluating AI outputs across combinatorics model research review reasoning, calculations, and research workflows. Reviewers grade derivations and assumptions, reproduce key results, and document the correct method...Hourly payFor contractorsRemote work- Surgical Planning Safety Evaluator is a remote evaluation track for reviewing surgical planning safety evaluation prompts... ...cases, and write the kind of structured feedback the modeling team can use to retrain. AI data reviewers help turn surgical planning safety evaluation...Remote job
$15 - $20 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...external tools. Generate high-quality human evaluation data by identifying response strengths, areas... ..., and completeness of responses. Ensure model responses align with expected...Contract workSummer workRemote work- AI Trainer Jobs is seeking an Illustration Quality Evaluator for a remote contractor role. Review illustration quality evaluation prompts and responses against a published rubric, compare paired outputs, and provide structured feedback to support retraining efforts. You...Remote jobPart timeFor contractors
- AI Trainer Jobs is seeking a remote contractor to evaluate people ops / recruiting prompts and responses against a evolving quality rubric. You will compare model outputs, label edge cases, and provide structured feedback for retraining. Responsibilities include evaluating...Remote jobFor contractors
- Receipt and Invoice Understanding Model Evaluator is a remote evaluation track for reviewing receipt and invoice understanding model evaluation... ...of structured feedback the modeling team can use to retrain. AI data reviewers help turn receipt and invoice understanding...Hourly payFor contractorsRemote work10 hours per week
- We’re seeking a future team member for the role of SVP - Model Risk Management AI, Wealth and Investment to join our Model Risk team. This role... ...regulatory expectations.Advise senior business, risk and Engineering stakeholders on model risk issues and emerging...WorldwideFlexible hours
The Bank of New York Mellon
New York, NY2 days ago $106.9k - $147k
...caring communityHumana is seeking a Senior DevOps Platform Engineer to design, implement, and support... ...solutions using PowerShell, Bash, Python, and AI-assisted engineering tools.Ensure HIPAA... ...with DORA metrics and DevOps models.Experience integrating automated testing...Full timeTemporary workApprenticeshipWork at officeRemote workHome officeHumana
New York, NY19 hours ago- ...seeking a Latin Bilingual Expert for a remote evaluation track. Reviewers compare paired outputs... ...generate structured feedback to retrain models. Responsibilities include evaluating... ...reliable async availability for at least 10 hours per week. #J-18808-Ljbffr AI Trainer JobsFor contractorsRemote work10 hours per week
- ...platforms with our developer community, driving adoption of modern DevOps tooling and AI-powered workflows like GitHub Copilot. You'll blend... ...skills to create engaging content, host trainings, and empower engineers to build faster and smarter.At EPAM, you'll work on cutting...
EPAM Systems
New York, NY4 days ago - ...About the role Swahili Evaluation AI Evaluator is a remote evaluation track for reviewing swahili generalist evaluation prompts and responses... ...edge cases, and write the kind of structured feedback the modeling team can use to retrain. Category: Frontier Model Evaluation...Hourly payFor contractorsRemote work10 hours per week
AI Trainer Jobs
New York, NY5 days ago - ...Our client is an AI-powered Data Analytics plaform for the enterprise commerce industry. They are based... ..., NY and are currently staffing in their Software Engineering Team. They are currently seeking a Senior DevOps / Infrastructure Engineer to help build and scale...Immediate start
The Cypress Group
New York, NY3 days ago - ...Obsidian is hiring expert Evaluators in real estate, hospitality, and events to review AI-generated work products for accuracy and quality. This is a remote position requiring strong expertise in the relevant domains. Ideal candidates should have over 5 years of professional...Work at officeRemote work
Obsidian
New York, NY2 days ago $100k - $140k
...to the Board and protecting the Firm from exposure to losses as a result of credit, market, liquidity, operational, model and other risks.Associate - AI Governance & ReportingPosition SummaryThe Associate will support the delivery of AI-enabled solutions across Model Risk...Temporary workMorgan Stanley
New York, NY5 days ago- MODEL RISK MANAGEMENT (MRM)The Model Risk Management (MRM) group is a multidisciplinary... ...Goldman Sachs is seeking a highly motivated AI Model Risk Vice President to join our... ...Computer Science, Mathematics, Physics, Engineering, or a closely related quantitative field....Work experience placement
Goldman Sachs
New York, NY1 day ago $130k - $160k
...PropositionAs Lead, Operating Model Design, you will serve as an internal... ...for Global Technology’s AI-First operating model. You... ...Enterprise DevSecOps, Platform Engineering, and Enterprise Architecture.... ...to incorporate field feedback, evaluate model effectiveness, and align...Full timeTemporary workWork experience placementWork at officeLocal areaRelocation package3 days per weekMetropolitan Life Insurance Company
New York, NY1 day ago$131.75k - $178.25k
...professional achievement and career growth. The Senior Infrastructure DevOps Engineer, under the direction of the Senior Manager of Infrastructure,... ...Apps.Experience with managing O365 services.Experience with AI technologies such as Copilot or Claude.Knowledge of container...Full timeWork experience placementRemote workWorldwideWilson Sonsini Goodrich & Rosati
New York, NY4 days ago$116.2k - $229.1k
Position Summary AI & Engineering/EaaS - DevOps Engineer IIIBuild and scale modern DevOps capabilities that help clients accelerate delivery... ...engineering processes and define target-state DevOps operating models, tooling, and workflowsBuilding and maintaining cloud...Local areaDeloitte
New York, NY1 day ago- ...About The Role The DevOps Engineer builds and operates the infrastructure that runs production services at scale. The role spans cloud architecture, deployment automation, observability, incident response, and reliability engineering across containerized workloads and...
Evlo AI
New York, NY1 day ago - ...are seeking a hands-on Cloud Engineer GCP Cloud to own the end-to-end... ...foundation powering our Agentic AI Solution Platform.Key... ...Active/Active-Passive deployment models.Security Identity ManagementDesign... ...modules and automated IaC pipelines.DevOps & Scripting: Proficient in...Work experience placement
Virtusa
New York, NY2 days ago $100k - $160k
...system. Today it is the platform that drives AI adoption across the enterprise.... ...there, WalkMe is the place for you!As a DevOps Engineer at WalkMe, you'll join our Cloud Engineering... ...maintain smooth operations.Participate in evaluating and integrating new technologies to...Full timeWork experience placementLocal areaImmediate startRemote workWalkMe
New York, NY3 days ago$96.3k - $260.1k
...DetailsAbout SalesforceSalesforce is the #1 AI CRM, where humans with agents drive... ...CAC/PIV.We are looking for an experienced DevOps engineer to join a team of highly skilled Systems... ...accurately and reliably.Critically evaluate code (Human or AI-generated) for correctness...Full timeWork experience placementLocal areaSalesforce
New York, NY3 days ago- ...il territorio nazionale, ricerca per importante cliente un/una DevOps Engineer Sede: Remoto Requisiti Almeno 3 anni di esperienza nel ruolo... ...finalità di selezioni del personale per esigenze attuali e future ai sensi del DL 196 del 20/06/2003 e dell Art. 13 GDPR 679/16....Remote work
$128k - $150k
...Additional information on the firm can be found at Role:**As a DevOps Engineer, you will play a crucial role in bridging the gap between... ...Apply automation, scripting, and emerging technologies, including AI-enabled engineering capabilities, to improve DevOps workflows...Work experience placementHeidrick & Struggles
New York, NY4 days ago$128k - $176k
...research confirms Lyra’s transformative care model helps people recover twice as fast and... ...Empower, the only fully integrated, AI-powered platform combining the highest-quality... ...We’re looking for an exceptional Sr. DevOps Engineer to join our team. As a Lyra Sr. DevOps...Full timeRemote workLever
New York, NY5 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator. Be the first to apply!
Related searches
- devops cloud engineer New York, NY
- devops engineer New York, NY
- senior devops engineer remote New York, NY
- devops engineer contract New York, NY
- big data devops engineer New York, NY
- devops aws developer (remote) New York, NY
- devops engineer sre New York, NY
- devops engineer remote New York, NY
- devops engineer full time New York, NY
- senior devops cloud engineer New York, NY