DevOps Engineer - AI Model Evaluator
$85 per hourMercor
Job Description
Job Description
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
- Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
- Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , observability , and infrastructure automation .
- Identify bugs, edge cases, reliability issues, and failure modes.
- Compare outputs from multiple frontier models and assess their strengths and weaknesses.
- Apply professional engineering judgment to realistic infrastructure engineering scenarios.
- Work independently and asynchronously to meet sprint-based project deadlines.
Qualifications
Must-Have
- 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
- Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling .
- Regular use of AI coding agents such as Cursor, Claude Code, Codex, Windsurf, Gemini CLI, or similar tools.
- Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
- Experience supporting production-scale systems.
Compensation & Legal
- $400 per accepted task .
- Compensation is tied to accepted work.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on us.fitly.work
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator in New York, NY vacancy
- Dorado is seeking an experienced Investment Banking SME to support the development, evaluation, and improvement of advanced AI models in finance. You will assess AI-generated analyses for accuracy, reasoned judgments, and data integrity, guiding model refinements and prompts...SuggestedRemote job
- Obsidian is seeking Insurance domain SMEs to join a cutting-edge AI training program. You will evaluate AI model outputs against real underwriting practice and rubrics, guiding research and engineering teams to close knowledge gaps in underwriting, claims, and risk assessment...Suggested
$80 per hour
...and technical talent with leading AI research labs. Headquartered in... ...Jack Dorsey . Position: Data Engineer (Coding Agent Experience) Type:... ...AI coding agents to complete and evaluate complex data engineering tasks. Review model-generated implementations...SuggestedContract workSummer workRemote workMercor
New York, NY4 days ago$20 per hour
Feedinkoo is looking for a Web Developer/Designer to enhance AI models by evaluating design work, including interfaces and user experiences. This role involves reviewing AI‑generated visuals and providing feedback to improve users' experience with AI tools. Working remotely...SuggestedRemote job$20 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...external tools. Generate high-quality human evaluation data by identifying response strengths, areas... ..., and completeness of responses. Ensure model responses align with expected...SuggestedRemote jobPart timeSummer work- Mercor is seeking a Generalist who can operate in English and Punjabi. This contract, remote position focuses on evaluating AI outputs and supporting model evaluation tasks. You will conduct fact-checking, assess reasoning, clarity, tone and completeness, and provide actionable...Remote jobContract work
- ...is seeking a Generalist to perform English and Tamil language evaluation tasks on a remote contract basis. You will conduct rigorous fact... ...data that highlights strengths, weaknesses, and inaccuracies in model responses. The role requires strong English writing, Tamil...Remote jobContract work
- Screenshot Understanding Model Evaluator is a remote evaluation track for reviewing screenshot understanding model evaluation prompts and responses... ...the modeling team can use to retrain. Why this role matters AI data reviewers help turn screenshot understanding model...Hourly payFor contractorsRemote work10 hours per week
- We’re seeking a future team member for the role of SVP - Model Risk Management AI, Wealth and Investment to join our Model Risk team. This role... ...regulatory expectations.Advise senior business, risk and Engineering stakeholders on model risk issues and emerging...WorldwideFlexible hours
- ...ServicesSelling Points Shape the future of AI infrastructure and DevOps at a leading organization.... ...operations.Job DescriptionSenior DevOps Engineer OverviewLead the design and optimization... ...scalable DevOps pipelines for model lifecycle automation and deployment in...
- ...platforms with our developer community, driving adoption of modern DevOps tooling and AI-powered workflows like GitHub Copilot. You'll blend... ...skills to create engaging content, host trainings, and empower engineers to build faster and smarter.At EPAM, you'll work on cutting...
$130k - $160k
...PropositionAs Lead, Operating Model Design, you will serve as an internal... ...for Global Technology’s AI-First operating model. You... ...Enterprise DevSecOps, Platform Engineering, and Enterprise Architecture.... ...to incorporate field feedback, evaluate model effectiveness, and align...Full timeTemporary workWork experience placementWork at officeLocal areaRelocation package3 days per week- MODEL RISK MANAGEMENT (MRM)The Model Risk Management (MRM) group is a multidisciplinary... ...Goldman Sachs is seeking a highly motivated AI Model Risk Vice President to join our... ...Computer Science, Mathematics, Physics, Engineering, or a closely related quantitative field....Work experience placement
$170k - $190k
As a Senior DevOps Engineer, you will be a critical member of our infrastructure team based in New York, providing earlier-hours coverage and... ...Google Cloud Platform (GCP) infrastructure as we expand our AI capabilities.This role requires 3 days in-office (Tuesday, Wednesday...Work at officeRemote workFlexible hours- ...DevOps EngineerWe're building the infrastructure layer for agentic web interaction at scale... ...(RAG) and real-time reasoning in AI systems. By connecting LLMs to high-quality... ...capacity planningWorking closely with a small engineering team — you'd own infra, not a slice of...
Tavily Inc
New York, NY1 day ago - ...City-based organization is building an agentic AI platform transforming financial services. As a Cloud Infrastructure Engineer, you will design and manage scalable, secure... ...to cutting-edge AI workloads, modern DevOps practices, and real-world impact on enterprise...Full timeRemote work
- ...Obsidian is hiring expert Evaluators in real estate, hospitality, and events to review AI-generated work products for accuracy and quality. This is a remote position requiring strong expertise in the relevant domains. Ideal candidates should have over 5 years of professional...Work at officeRemote work
Obsidian
New York, NY2 days ago $14.5 per hour
...Join to apply for the AI Web Search Evaluator role at Welo Data Welo Data works with technology... ...and scalable to supercharge their AI models. As a Welocalize brand, Welo Data leverages... ...in improving the quality of search engine results, ensuring users find the most...Hourly payPart timeImmediate startRemote workWork from home10 hours per weekFlexible hoursWelo Data
New York, NY3 days ago$11.5 per hour
...an Online Task Contributor. In this role, you will evaluate and provide feedback on content to enhance search engine results and quality. No prior experience is needed... ...completion, with a supportive community of contributors involved in AI advancements. #J-18808-LjbffrHourly payPart timeRemote workUniversity of Delaware
New York, NY16 hours ago- ...DevOps EngineerTitle of Role: DevOps EngineerLocation: New York, New York, onsiteCompany... ...dynamic company that is at the forefront of AI technology, focusing on automation and... ...fast-paced, results-driven culture where engineers are empowered to innovate, take ownership...
Recruiting from Scratch
New York, NY2 days ago $180k - $230k
SVP, MLOps / DevOps Engineer - Full Time - Hybrid We’re partnering with our client, a fast-growing... ...scale the infrastructure behind their AI and machine learning platforms. This is... ...engineering, data, and cloud; helping bring AI/ML models into production in a secure, scalable,...Full timeRemote work$180k - $225k
As a Software Engineer on the ML Infrastructure team, you will design... ...engineers to integrate and optimize models for production and research... ...to ensure a fair and thorough evaluation of all applicants.About Us:At... ...is to develop reliable AI systems for the world's most important...Full time$96.3k - $260.1k
...DetailsAbout SalesforceSalesforce is the #1 AI CRM, where humans with agents drive... ...CAC/PIV.We are looking for an experienced DevOps engineer to join a team of highly skilled Systems... ...accurately and reliably.Critically evaluate code (Human or AI-generated) for correctness...Full timeWork experience placementLocal area$180k - $230k
...creative, and intellectually curious MLOps/DevOps Engineers with deep expertise in machine learning... ...‑grade MLOps platforms, including model lifecycle automation, observability, and... ...monitoring).Enable production workloads for AI/ML and Generative AI systems, including...Full timeWork at officeRemote work$145k - $165k
...for. About the RoleThe Senior Engineer, Cybersecurity is a hands-on contributor... ...security reviews, threat modeling, and risk assessments (code,... ..., vulnerability management, AI security, network security,... ...Qualifications:Security/Compliance or DevOps certificationsExperience with...Local areaFlexible hoursShift work$200k - $250k
...Senior DevOps EngineerNew YorkFounded 20 years ago and headquartered... ...for a senior level DevOps Engineer to join a small, high-impact team... ...help build a sustainable support model as the platform scales.... ...storage systems.Thoughtful use of AI coding assistants and interest...WorldwideDV Trading
New York, NY16 hours ago$180k - $240k
...DevOps EngineerWhere Medicine Meets IntelligenceDoctronic is the first AI legally authorized to practice medicine. We're processing millions... ...'re looking for a DevOps Engineer to own our infrastructure. We'... ...services and Large Language Models (LLMs).Support scalable inference...Flexible hoursDoctronic
New York, NY1 day ago- ...Knock DevOps EngineerKnock is on a mission to help products communicate with their users... ...About the RoleWe're looking for a DevOps engineer to join our small but growing platform team... ...for us, our future selves, and our AI companions.A Note on AI at KnockWe're a team...Remote work
Knock
New York, NY2 days ago - ...providing independent assurance and evaluating the company's risk management... ...for data scientists and AI developers who will power our... .... You will be deploying your engineering, data analytics and data... ...ResponsibilitiesProficiency in frameworks for auditing models, including criteria like...
$118.98k - $195.47k
...individual to join our team as a Finance Model & AI Solutions Lead.The colleague in this... ...capabilities that automatically create and evaluate multiple planning scenariosImplement AI... ...team members on AI tools, prompt engineering, and best practices for human-AI collaborationYou...Full timeH1bWork at officeVisa sponsorshipWork visaFlexible hours3 days per week
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator. Be the first to apply!
Related searches
- devops engineer contract New York, NY
- senior devops cloud engineer New York, NY
- senior devops engineer New York, NY
- devops engineer sre New York, NY
- devops cloud engineer New York, NY
- devops engineer New York, NY
- devops engineer remote New York, NY
- senior devops engineer remote New York, NY
- devops aws developer (remote) New York, NY
- devops engineer full time New York, NY