DevOps Engineer - AI Model Evaluator
$85 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
- Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
- Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , observability , and infrastructure automation .
- Identify bugs, edge cases, reliability issues, and failure modes in model outputs.
- Compare outputs from multiple frontier models to assess their strengths and weaknesses.
- Apply professional engineering judgment to realistic infrastructure engineering scenarios.
Qualifications
Must-Have
- 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
- Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling.
- Regular use of AI coding agents such as Cursor, Claude Code, Codex, Windsurf, Gemini CLI, or similar tools.
- Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
- Experience supporting production-scale systems.
Compensation & Legal
- $400 per accepted task
- Compensation tied to accepted work.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on ziprecruiter.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator in New York, NY vacancy
$40 per hour
...data annotation company is seeking a Statistician to join their team. This remote role involves training AI models by posing complex mathematical problems, evaluating outputs, and assessing the model's performance. Candidates must be detail-oriented and proficient in...SuggestedHourly payRemote workDataAnnotation
New York, NY2 days ago$40 per hour
A cutting-edge AI company is seeking experienced quantitative professionals to evaluate AI-generated work and help advance AI development. This fully remote role allows... ...to reason about data. Join us to help shape the future of AI models! #J-18808-Ljbffr DataAnnotationSuggestedRemote jobHourly pay- A leading AI development company is seeking experienced quantitative professionals for remote work evaluating AI-generated quantitative analysis. Ideal candidates will have a robust background in fields like data science, economics, or biostatistics, with at least 2 years...SuggestedRemote job
$40 per hour
A dynamic AI development company is seeking experienced quantitative professionals in Georgia to evaluate AI-generated work and provide vital feedback. This role is fully remote, allowing candidates from various countries to contribute flexibly. Ideal applicants will have...SuggestedRemote jobHourly pay$40 per hour
An innovative AI firm is seeking a Postdoctoral Biology Associate to join their team in evaluating and improving AI chatbots' logic based on biology. Applicants must hold or be in progress for a PhD in Biology or a related field, with a strong understanding of cell biology...SuggestedRemote jobHourly payFlexible hours$60 per hour
A leading AI development company is seeking quantitative professionals to evaluate AI-generated outputs and solve complex problems. This fully remote role allows flexibility and offers competitive pay of up to $60 USD/hour. Ideal candidates should have at least 2 years...Remote workFlexible hoursDataAnnotation
New York, NY2 days ago- A leading AI training company is seeking medical experts to evaluate AI chatbots' responses to complex healthcare problems in a REMOTE position. Applicants need to hold a medical degree or be in-progress towards one. Responsibilities include ensuring medical accuracy of...Remote jobHourly pay
- Welocalize is seeking a Data Quality Associate in New York to evaluate AI model outputs and provide structured feedback. This full-time position requires native-level language proficiency and a university degree. You will contribute to improving evaluation frameworks while...Full timeRemote work
$40 per hour
...cybersecurity professionals to join our team to help train AI models. In this role, you will evaluate AI-generated security content, solve technical... ...penetration testing, red teaming, incident response, detection engineering, DFIR, malware analysis, threat intelligence, or...Hourly payFull timePart timeRemote workDataAnnotation
New York, NY2 days ago- ...leading cybersecurity firm is seeking experienced professionals to evaluate AI-generated cybersecurity content and solve technical security problems. You will play a significant role in training AI models, providing critical feedback, and improving system accuracy. This...Remote jobFlexible hours
$20 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...external tools. Generate high-quality human evaluation data by identifying response strengths, areas... ..., and completeness of responses. Ensure model responses align with expected...Remote jobContract workPart timeSummer work- We’re seeking a future team member for the role of SVP - Model Risk Management AI, Wealth and Investment to join our Model Risk team. This role... ...regulatory expectations.Advise senior business, risk and Engineering stakeholders on model risk issues and emerging...WorldwideFlexible hours
- MODEL RISK MANAGEMENT (MRM)The Model Risk Management (MRM) group is a multidisciplinary... ...Goldman Sachs is seeking a highly motivated AI Model Risk Vice President to join our... ...Computer Science, Mathematics, Physics, Engineering, or a closely related quantitative field....Work experience placement
$124k - $177k
...Artificial Intelligence & Data (AI&D) organization, you will:... ...stakeholders to ensure proper modeling processes are followed during... ...into practical controls and evaluations (e.g., bias testing, stress testing... ...across data science, engineering, and product teams.• Stay up-...Local area3 days per week$14.5 per hour
A technology company is seeking an AI Web Search Evaluator to enhance the quality of search engine results. This flexible, remote, part-time role focuses on analyzing search performance and providing feedback to improve algorithms. Ideal candidates will have strong analytical...Hourly payPart timeRemote workFlexible hoursWelo Data
New York, NY2 days ago$20 per hour
A tech company is seeking a Receptionist to assist in training AI models. This role involves evaluating the logic of AI chatbots and ensuring quality in their outputs. Applicants should be fluent in English, detail-oriented, and possess strong writing skills. This position...Hourly payRemote workFlexible hoursDataAnnotation
New York, NY3 days ago- ...Obsidian is hiring expert Evaluators in real estate, hospitality, and events to review AI-generated work products for accuracy and quality. This is a remote position requiring strong expertise in the relevant domains. Ideal candidates should have over 5 years of professional...Work at officeRemote work
Obsidian
New York, NY1 day ago - ...MERIT Beauty is seeking Turkish-speaking annotators for a contract role evaluating AI-generated content. You will review content for coherence, consistency, and alignment with real-world expectations in Turkish. The ideal candidates are native or fluent Turkish speakers...Contract workTemporary workImmediate startRemote work
MERIT Beauty
New York, NY3 days ago $20 per hour
...A tech company specializing in AI is seeking an Experience Designer to evaluate and enhance AI-generated UI/UX designs. This remote position allows for flexible scheduling and offers competitive pay rates starting at $20/hour for general projects and $40/hour for design...Remote workFlexible hoursDataAnnotation
New York, NY3 days ago$40 per hour
...A leading cybersecurity firm is seeking experienced professionals to evaluate AI-generated security content and solve cybersecurity problems. This role, which can be part-time or full-time, offers flexible hours and competitive pay starting from $40+ per hour. Ideal candidates...Hourly payFull timePart timeRemote workFlexible hoursDataAnnotation
New York, NY2 days ago- A leading AI company is seeking detail-oriented linguists with native Turkish fluency for a remote freelance opportunity. This role will focus on AI-related projects that involve prompt evaluation and multimedia content understanding. Ideal candidates will have a strong...FreelanceRemote work
Lilt
New York, NY5 days ago - ...finance experts, the full-time Remote AI Research Evaluator will assess AI-generated financial content... ...relevant questions, and evaluate model responses in a flexible contract role.... ...stages) in Quantitative Finance, Financial Engineering, Financial Mathematics, or a closely...Full timeContract workRemote workFlexible hours
Freelance Labs Inc. d/b/a Braintrust
New York, NY2 days ago $140k - $260k
...activity across blockchains?Elliptic is looking for a Senior DevOps Engineer to join our DevOps team. This is a hands-on technical role, working... ...crypto activity. Current major projects include delivering AI Gateway and supporting LangChain/LangGraph infrastructure for...Remote workHome office$228.7k - $343.1k
...at enormous scale, and one bad model can mean millions in credit... ...you apply to models applies to AI. We build the tooling that lets... ...validate at scale, so you critically evaluate what it produces and own the... .... Solid software and data engineering: production-quality Python,...Remote jobFull timeLocal areaShift workBlock
New York, NY1 day ago$180k - $225k
As a Software Engineer on the ML Infrastructure team, you will design... ...engineers to integrate and optimize models for production and research... ...to ensure a fair and thorough evaluation of all applicants.About Us:At... ...is to develop reliable AI systems for the world's most important...Full time- ...City-based organization is building an agentic AI platform transforming financial services. As a Cloud Infrastructure Engineer, you will design and manage scalable, secure... ...to cutting-edge AI workloads, modern DevOps practices, and real-world impact on enterprise...Full timeRemote work
$170k - $190k
As a Senior DevOps Engineer, you will be a critical member of our infrastructure team based in New York, providing earlier-hours coverage and... ...Google Cloud Platform (GCP) infrastructure as we expand our AI capabilities.This role requires 3 days in-office (Tuesday, Wednesday...Work at officeRemote workFlexible hours- ...We are seeking a self-driven Junior DevOps Engineer to contribute to the development, deployment, and maintenance of data and AI products that support positive social impact around the world. You will support the maintenance of the cloud infrastructure that powers our...
Patrick J. McGovern Foundation - Remote, United States
New York, NY5 days ago $180k - $230k
SVP, MLOps / DevOps Engineer - Full Time - Hybrid We’re partnering with our client, a fast-growing... ...scale the infrastructure behind their AI and machine learning platforms. This is... ...engineering, data, and cloud; helping bring AI/ML models into production in a secure, scalable,...Full timeRemote work- ...Medium is seeking a Junior DevOps Engineer to contribute to the development of AI and data products for social impact. You will maintain cloud infrastructure and support application development while working with nonprofit partners. The ideal candidate has a Bachelor...Full timeRemote work
Medium
New York, NY3 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator. Be the first to apply!
Related searches
- big data devops engineer New York, NY
- devops cloud engineer New York, NY
- devops engineer sre New York, NY
- devops engineer remote New York, NY
- devops engineer azure New York, NY
- senior devops engineer remote New York, NY
- senior devops engineer New York, NY
- devops engineer New York, NY
- senior devops cloud engineer New York, NY
- devops engineer full time New York, NY