Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Benchmark & Red-Teaming Specialist

Turing

Research and build benchmark tasks to expose AI model weaknesses. About Turing: Based in San Francisco, California, Turing is the world’s leading research accelerator for frontier AI labs and a trusted partner for global enterprises deploying advanced AI systems. Turing supports customers in two ways: first, by accelerating frontier research with high-quality data, advanced training pipelines, plus top AI researchers who specialize in coding, reasoning, STEM, multilinguality, multimodality, and agents; and second, by applying that expertise to help enterprises transform AI from proof of concept into proprietary intelligence with systems that perform reliably, deliver measurable impact, and drive lasting results on the P&L. Role Overview We are seeking an analytical researcher and red-teaming specialist to build evidence-based web-research benchmark tasks that reliably expose weaknesses in state-of-the-art (SOTA) AI models (e.g., Claude Opus 5 Max). In this role, you will discover hard-to-surface facts, build precise research tasks around them, test them directly against leading LLMs, and diagnose the exact failure mechanisms in the model's search and reasoning paths. Your work will directly shape AI evaluation frameworks, ensuring next-generation frontier models are tested against rigorous, non-trivial real-world research scenarios. Job Requirements: Core Experience: 1+ years in investigative research, fact-checking, AI benchmark task authoring, red-teaming, or technical QA review. Benchmark Familiarity: Proven experience authoring, testing, or reviewing tasks for AI benchmarks such as GAIA, BrowseComp, SimpleQA, SWE-bench, Terminal-Bench, or Humanity’s Last Exam (HLE) . Analytical & OSINT Skills: Intermediate-to-advanced web search and Open Source Intelligence (OSINT) skills; ability to navigate academic archives, local records, or obscure databases. Model Trajectory Debugging: Strong analytical capability to inspect model reasoning chains, search logs, and source selection to diagnose root causes of failure. Precision Writing: Ability to author clear, unambiguous benchmark prompts with bulletproof, primary-sourced answer keys. Job Responsibilities: Obscure Fact Discovery: Deeply research niche topics to identify factual ground truths that cannot be surfaced via simple, direct search queries. Source Verification: Rigorously verify facts against primary or highly credible sources, cross-check for contradictory evidence, and document exact source URLs. Task Construction: Draft clear, answerable benchmark tasks designed specifically to expose structural weaknesses in model research strategies. Red-Teaming & Model Testing: Execute tasks against designated SOTA models to confirm they reliably trigger intended model failures. Failure Diagnosis & Debugging: Trace model run logs, search trajectories, and intermediate outputs to pinpoint exactly where and why the model failed (distinguishing genuine model limitations from ambiguous prompts). Pattern Scaling & Review: Apply validated "model-breaking" research patterns across new domains and, for strong performers, review tasks authored by peer trainers. Strong Plus (Preferred Qualifications) Bachelor or higher degree from relevant engineering domain is a must for this position. Prior experience specifically designing prompt-injection or reasoning-trap tasks that break SOTA models. Advanced OSINT capabilities or published investigative/academic research demonstrating deep source verification. Niche domain expertise in areas rich with hard-to-find records (e.g., local government filings, regional sports history, legal archives, historical patents). Experience serving as a lead reviewer, benchmark auditor, or quality owner on AI evaluation projects. Offer Details: Commitments Required: Upto 40 hours per week with 4 hours of overlap with PST. Engagement type: Contractor Engagement Length: upto 4 weeks Skills: LLM, Quality Management, QA #J-18808-Ljbffr Turing

Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the AI Benchmark & Red-Teaming Specialist in New York, NY vacancy
  • $60 - $90 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel ,...  ..., and Jack Dorsey . Position: LLM Red Team Specialist — Failure Modes & Edge Cases Type:... 
    Suggested
    Contract work
    Summer work
    Remote work

    Mercor

    New York, NY
    10 days ago
  • Mercor is assembling a panel of nuclear domain experts to red-team frontier AI models. The goal is to test whether a model can correctly judge the misuse potential of a technical request — answering legitimate questions fully while refusing genuinely dangerous ones. You... 
    Suggested

    Neon

    New York, NY
    3 days ago
  • $169k - $228.6k

     ...pioneering the future of generative AI and eager to help the world's...  ...Financial Services (GFS) team seeks an experienced...  ...including model performance benchmarking, output quality assessment, and...  ...focused, high-calibre group of specialist Solutions Architects supporting... 
    Suggested
    Flexible hours

    AmazonWebServices

    New York, NY
    1 day ago
  • $65k - $70k

     ...Careers REMOTE Operations Opportunity in Insurance Industry Team Success Specialist, Highland Capital Brokerage Location(s): Birmingham/ REMOTE...  ...organizational challenges that impact the team. Raising red flags wherever the business process needs correction to ensure... 
    Suggested
    Full time
    Remote work
    Shift work

    Advisor Group

    New York, NY
    2 days ago
  • $45.36k - $62.37k

     ...professionals do their best work. Join a team using innovation and insight to make a...  ...Description:In Patient Facility Homecare Specialist (Kaiser Team) - Remote 1:00pm - 10:00pm...  ...competitive pay and benefits, Solventum regularly benchmarks with other companies that are comparable... 
    Suggested
    Remote job
    H1b
    Work at office
    Relocation package
    Flexible hours
    Shift work

    Solventum

    New York, NY
    22 hours ago
  • $54.77k

    Online Compensation Benchmarking Specialist Location: Remote Salary: USD 54,774 per year Job Description: We seek a detail-oriented and analytical Online Compensation Benchmarking Specialist to join our team. In this remote position, you will evaluate and compare compensation... 
    Work at office
    Remote work
    Flexible hours

    Naukri Mitra

    New York, NY
    2 days ago
  • $35 - $49 per hour

    Review how frontier AI models handle chemical, biological, radiological and nuclear questions, and decide what belongs in a safe...  ...response, incident command or exercise experience. Prior AI safety, red-teaming, policy review or rubric-based evaluation work. What success... 
    Hourly pay
    Contract work
    For contractors
    Remote work
    Shift work

    Sovrano AI

    New York, NY
    5 days ago
  • $61 - $77 per hour

    Applied Mathematics Benchmark Specialist is a remote review track for evaluating AI outputs across mathematics reasoning, calculations, and research workflows. Reviewers...  ..., and document the correct method so the modeling team can train on it. Category: Scientific AI & Domain... 
    For contractors
    Remote work
    10 hours per week

    Auraone

    New York, NY
    3 days ago
  • $77 - $98 per hour

     ...technical talent with leading AI research labs. Headquartered in...  ..., our investors include Benchmark , General Catalyst , Peter...  ...Economics & Finance Benchmark Specialist Type: Contract Compensation...  ...****@*****.*** PS: Our team reviews applications daily.... 
    Contract work
    Summer work
    Remote work

    Mercor

    New York, NY
    5 days ago
  • $55k - $60k

     ...health clinicians, and workforce development specialists. Together, these services support our...  ...: Employment Specialist  Location : Red Hook (Brooklyn) Reports To : Senior Employment...  ...employment outcomes. Participate in team meetings, program meetings, and other... 
    Permanent employment
    Full time
    Part time
    Casual work
    Placement year
    Local area
    Weekend work
    Weekday work

    Jericho Project

    New York, NY
    9 days ago
  • $45 - $70 per hour

     ...elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam...  ...reach out to: ****@*****.*** PS: Our team reviews applications daily. Please complete... 
    Contract work
    Summer work
    Remote work
    Weekday work

    Mercor

    New York, NY
    13 days ago
  • $77 - $98 per hour

     ...technical talent with leading AI research labs. Headquartered in...  ..., our investors include Benchmark , General Catalyst , Peter...  ...Economics & Finance Benchmark Specialist Type: Contract Compensation...  ...****@*****.*** PS: Our team reviews applications daily.... 
    Contract work
    Summer work
    Remote work

    Mercor

    New York, NY
    7 days ago
  • $50 - $63 per hour

     ...technical talent with leading AI research labs. Headquartered in...  ..., our investors include Benchmark , General Catalyst , Peter...  ...Applied Philosophy Benchmark Specialist Type: Contract Compensation...  ...****@*****.*** PS: Our team reviews applications daily.... 
    Contract work
    Summer work
    Remote work

    Mercor

    New York, NY
    11 days ago
  • $70 - $110 per hour

     ...technical talent with leading AI research labs. Headquartered...  ...Francisco, our investors include Benchmark , General Catalyst ,...  ...Position: Project Management Specialists Type: Contract Compensation...  ...with AI research teams to improve AI model performance... 
    Hourly pay
    Contract work
    Summer work
    Immediate start
    Remote work

    Mercor

    New York, NY
    a month ago
  • $75 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam...  ...provide clear rationales. Preferred ~ Red-teaming experience in radiological safety.... 
    Contract work
    Summer work
    Remote work

    Mercor

    New York, NY
    6 hours ago
  •  ...information assurance, or a related field is preferred. Relevant certifications or practical experience in security operations, red or blue team work, or cyber policy are useful indicators of fit. Benefits and work setup Freelance contract, remote. Listed pay range is $6... 
    Hourly pay
    Contract work
    For contractors
    Freelance
    Remote work

    agency

    New York, NY
    3 days ago
  •  ...cybersecurity expert eager to shape the future of AI? Large-scale language models are evolving...  ...of AI. We're looking for cybersecurity specialists who live and breathe network security,...  ...hands-on experience in SOC environments, red team/blue team operations, or cyber policy... 
    Hourly pay
    Contract work
    Remote work

    Invisible Technologies Inc.

    New York, NY
    1 day ago
  • $207k - $275k

    CoreWeave is The Essential Cloud for AI. Built for pioneers by...  ...platform of technology, tools, and teams that enables innovators to...  ...a market-maker. As a Solution Specialist for AI Runtime Services, you open...  ..., technical playbooks, and benchmark narratives that help sales and... 
    Permanent employment
    Full time
    Temporary work
    Casual work
    Work at office
    Flexible hours

    CoreWeave

    New York, NY
    2 days ago
  •  ...About the Opportunity Our client, a leading AI research organization, is looking for...  ...architects and architectural visualization specialists to support AI training and evaluation....  ...advance will then participate in a 45-minute technical interview with the team.... 
    Afternoon shift

    Braintrust

    New York, NY
    6 hours ago
  • $65 - $75 per unit

     ...elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam...  ...ones. Collaborate with AI research teams to improve model judgment on misuse potential... 
    Contract work
    Summer work
    Remote work
    Flexible hours

    Mercor

    New York, NY
    6 hours ago
  • $45 - $55 per hour

     ...elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam...  ...clear, structured feedback to AI research teams to improve training data quality and... 
    Contract work
    Summer work
    Remote work

    Mercor

    New York, NY
    2 days ago
  •  ...founded in 2017 in Europe. We collect data where automation is not possible. Our teams support projects involving speech recording, transcription, image and video collection, mapping, and AI training across Europe and North America. Project Overview We are looking for... 
    Remote work
    Flexible hours

    TSMG Holding

    New York, NY
    2 days ago
  •  ...oriented transcription participants for a remote project supporting AI speech and language systems. You will listen to audio...  ...and the target language. Flexible remote work and a supportive project team are offered to help you meet deadlines and #J-18808-Ljbffr... 
    Remote work
    Flexible hours

    TSMG

    New York, NY
    8 hours ago
  •  ...Vietnamese Voice Actor - Freelance AI Trainer Project World Wide - Remote Are you an...  ...annotate errors, and collaborate with our team to refine prompts, evaluation methods, and...  .... Job title: Vietnamese Voice Acting Specialist – AI Trainer Employment type: Contract Workplace... 
    Hourly pay
    Contract work
    For contractors
    Freelance
    Remote work

    Meridial

    New York, NY
    1 day ago
  • $110k - $138k

     ...organizations design, build and scale intelligent AI, digital, cybersecurity, cloud and...  ...capabilities. With more than 14,000 team members and over 60 locations globally, WWT...  ...will you be doing? The Computer Vision Specialist is a member of WWT’s Global Solutions &... 
    Full time
    Work experience placement
    Remote work
    Flexible hours
    Shift work

    World Wide Technology

    New York, NY
    3 days ago
  • $100k - $125k

     ...Job Title: Senior IT Specialist Location: New York, NY (in-office) Salary: $100K – $125K...  ...seeks a Senior IT Specialist to join their Team Responsibilities: • Administer and...  ...platforms, accounting systems, and legal AI tools • A bachelor’s degree in a computer... 
    Work at office

    Addition Management

    New York, NY
    7 days ago
  • $31.82 - $36.77 per hour

     ...ambitious goal to become the benchmark of safety excellence across all...  ...As a Workplace Health & Safety Specialist (WHSS), you'll be at the...  ...Heart Association or American Red Cross, must obtain within 3 weeks...  ...Experience leading and managing a team, ability to analyze accident... 
    Hourly pay
    Flexible hours
    Shift work
    Night shift
    Weekend work

    Amazon

    New York, NY
    1 day ago
  • AssemblyAI is building the leading Voice AI infrastructure platform, and our go-to-market team is growing fast. We're looking for a GTM Recruiter to help us hire the Account Executives, BDRs, Customer Success, Forward Deployed Engineers, and Marketers who bring our models... 

    AssemblyAI

    New York, NY
    3 days ago
  • Welocalize is seeking a Vietnamese→English localization specialist to join our distributed team in the United States. You will manage translation and copy-editing tasks on assigned projects and support terminology management. Ideal candidates have 4+ years in translation... 
    Remote job

    Welocalize, Inc.

    New York, NY
    4 days ago
  •  ...seeking experienced Danish Linguistic Experts to join a cutting-edge AI research project in partnership with a leading research lab. You...  ...or NLP data work. Collaboration happens in a distributed, remote team, contributing to state-of-the-art AI research. #J-18808-Ljbffr... 
    Remote work

    Mercor

    New York, NY
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Benchmark & Red-Teaming Specialist. Be the first to apply!