AI Benchmark Researcher
$1,350 per unitBraintrust
Bring your expertise to the problems today’s most advanced AI agents still struggle to solve.
We’re looking for experienced professionals to develop challenging tasks for an AI benchmark. You’ll draw on problems from your own field and turn them into clearly defined challenges that an AI agent must solve from scratch in a Linux terminal.
What you’ll do
- Identify difficult, meaningful problems in your area of expertise.
- Translate those problems into tasks that can be solved in a Linux terminal.
- Design challenges that expose limitations in advanced AI agents. Each task is tested against three leading models, with three attempts per model, and must prove difficult enough that the models mostly fail.
Who we’re looking for
- A completed master’s degree or higher. Candidates with a bachelor’s degree, at least 10 years of relevant professional experience and a publication may also be considered.
- At least 10 years of professional experience in a technical domain.
- Expertise in coding, machine learning, systems, cybersecurity or hardware.
- At least one academic or professional publication.
- Practical comfort working in a Linux terminal.
- Currently based in the United States.
Project details
- Task-based project expected to run for at least seven weeks.
- Onboarding includes two courses and a graded assessment, taking approximately 90 minutes.
- A live onboarding webinar is also required.
Compensation
- Earn up to $1,350 per task , with opportunities for higher rates based on task quality and submission volume.
- Tasks take six to seven hours on average, although initial tasks may take longer as you get familiar with the process.
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the AI Benchmark Researcher in Austin, TX vacancy
- Google is inviting students to join its Student Researcher program, offering research collaborations on real-world, large-scale problems across multiple Google teams, including DeepMind and Google Research. Roles support flexible engagement timelines, with onsite and remote...SuggestedRemote jobFlexible hours
- CrowdStrike is seeking a self-starting Security Researcher to join our Counter Adversary Operations Team in the Austin area. You will track Latam-language eCrime actors, collect unique, timely intelligence from diverse sources including the deep and dark web, and deliver...Suggested
$56 - $63 per hour
...demonstrate impact and growth. Our Team Visa Research expands Visa’s technology research... ...to advance the next generation of AI for secure, intelligent, and scalable commerce... .... Build tools and frameworks to benchmark agent safety, robustness, and performance...SuggestedHourly payFull timeInternshipWork at officeLocal areaVISA
Austin, TX27 days ago- ...working alongside a global ecosystem of 250+ research groups. Our groundbreaking Loihi series... ...power the next generation of physical AI systems.We are seeking a Neuromorphic AI... ...artifacts including APIs, kernels, and benchmarking tools that enable collaborative...SuggestedFull timeInternshipWork at officeLocal areaImmediate startShift work
Intel
Austin, TX1 day ago - ...AI Research Scientist The AI Research Scientist will contribute to webAI's development of next-generation AI models and systems. In... ...inference strategies for distributed AI Develop and evaluate benchmarks, datasets, and experimental frameworks to test model...SuggestedLive outWork at officeLocal areaFlexible hours
G2 Venture Partners
Austin, TX3 days ago $105.05k - $152.35k
...UX Researcher Description - Job Summary The Design Systems Research and Governance organization is responsible for creating and evolving... ...Dscout HeyMarvin or similar solutions. Experience leveraging AI-enabled research tools and workflows to accelerate study...Full timeTemporary workLocal areaRemote workRelocationFlexible hoursShift workHP
Austin, TX2 days ago$123.8k - $185.7k
...production support of scalable data science, AI, and machine learning solutions that... ...and production-ready capabilities for benchmarking, similarity modeling, causal inference,... ...early-stage product, R&D, innovation, or research-oriented environments. Experience building...Remote workFlexible hoursJohn Deere India Pvt. Ltd.
Austin, TX1 day ago$15 - $18.5 per hour
A civic technology organization seeks a Researcher to explore election topics and candidate qualifications. Responsibilities include researching various election-related topics across the U.S., mentoring new researchers, and contributing to data collection processes. Candidates...Hourly payContract workRemote workHard Yaka
Austin, TX46 minutes ago$98.1k - $167.33k
...information, visit RESPONSIBILITIES Design, develop, and maintain interactive dashboards and reports using traditional or modern, AI-powered BI tools to support business decision-making across departments. Write and optimize complex SQL queries to extract, transform...Work at officeWorldwideRelocation package- VulnCheck, The Exploit Intelligence Company, is seeking a Senior Vulnerability Researcher for its Initial Access Intelligence group. You will drive original vulnerability research, reverse engineer firmware, and develop weaponized exploits across various OSes and devices...Remote job
- The University of Texas at Austin is seeking a Field Research Assistant to support invasive pasture mealybug studies in South Texas. The role involves field surveys, specimen collection, data entry, and collaboration with researchers at Brackenridge Field Laboratory and...
$68k
Job Title Geothermal Conformance and Flow Control Researcher Hiring Department Bureau of Economic Geology Position Open To All Applicants Weekly Scheduled Hours 40 FLSA Status Exempt from FLSA Earliest Start Date Immediately Position Duration Expected to Continue Until...For contractorsWork at officeLocal areaImmediate startAfternoon shift$15 - $18.5 per hour
Civitech in Austin, Texas is looking for a Researcher to support civic participation by researching various elections and candidate details. You’ll play a crucial role in providing accurate data and engaging with team initiatives aimed at progressive causes. This contractor...Remote jobFor contractorsFlexible hours- American Institutes for Research seeks a Principal Researcher to lead CMS Innovation Center evaluations within the Healthcare Innovations team. You will guide complex studies, design evaluation plans, and interact with federal clients to shape policy. The role requires...Remote job
$65k - $105k
Opposition Researcher | Political Consulting Firm | Austin, TX | Job ID: #3669697-1 Are you a strong researcher and writer with an interest in politics, public affairs, and investigative research? A political consulting firm in Austin, Texas, is seeking a Researcher to...$142.8k - $193.2k
...We are seeking an Applied Scientist to join our AI Security team which builds AI-based security analysis services that identify validate... ...coverage across Amazon. As an Applied Scientist you will research model design and implement machine learning and agentic systems...Full timeWork from homeFlexible hoursAmazon
Austin, TX2 days ago$105k - $130k
...for a challenging and rewarding career with a leading organization, come see what we can offer you! Position Biotechnology Researcher for Microbial Genetic/Synthetic Biology Engineering Location: Austin, Texas Security Clearance: Active Clearance or Eligible...For contractorsFlexible hoursSURVICE Engineering Company
Austin, TX28 days ago$60 per hour
Prolific is seeking Computer Science Specialists in Austin, TX, to train AI models and ensure data integrity. Ideal candidates possess at least a BSc in Computer Science and have strong technical literacy. Responsibilities include evaluating AI-generated responses and...Flexible hours- ...ABOUT WAY Way is building the AI, data, and infrastructure layer for the experience economy. As artificial intelligence automates labor and people reclaim time, the shift toward spending on experiences over things only accelerates. Way sits at the center of that...Work at officeShift work
Way.io
Austin, TX1 day ago $107k - $153k
A leading technology firm in the United States is seeking an Executive Recruiting Researcher to identify and engage leadership talent. You will develop search strategies, manage hiring processes, and collaborate with leadership recruiters to provide insights. The role...Full time$145k - $215.05k
...take care of ourselves, each other, and our communities.**Job Summary:**We’re looking for a curious and empathetic Staff Experience Researcher who is passionate about understanding user needs. Joining the Checkout team, you’ll play a vital role in uncovering deep insights...Full timeWork at officeLocal areaImmediate startFlexible hours- ...Support compliance with the Multi-Sector General Permit (MSGP) for Industrial Stormwater, including sector applicability evaluations, benchmark monitoring, corrective actions, and annual reporting. Develop and maintain Spill Prevention, Control, and Countermeasure (SPCC)...InternshipLocal area
ESE Partners
Austin, TX12 hours ago - ...graduate courses with emphasis on cost accounting, data analytics, and potential tax accounting expertise. The successful candidate will maintain an active research agenda, engage with the Austin business community, and contribute to #J-18808-Ljbffr St. Edward's UniversityFull time
- Visa Research offers a 12-week internship program where you will work on impactful AI projects to advance Visa's mission. You will build connections, strengthen communication and business skills, and gain hands-on experience in a dynamic, global environment. You'll attend...Internship
- ...science. We encourage candidates with strong interdisciplinary potential and joint appointments across departments for collaborative research. Successful candidates will pursue an active research program, teach graduate and undergraduate courses, and supervise students....
- ...invites applications for a full-time faculty position in CSTEM. The role emphasizes teaching undergraduate STEM courses and conducting research, with open-rank appointments available. We seek candidates specializing in computer science, statistics, applied mathematics, and...Full timeSummer work
$55 per hour
...Please submit your CV in English and indicate your level of English proficiency. Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not...Permanent employmentTemporary workPart timeFreelance10 hours per weekMind Rift
Austin, TX4 days ago- PayPal is seeking a curious and empathetic Staff Experience Researcher for the Checkout team. You’ll uncover deep insights that inform user-centered experiences across a consumer’s end-to-end purchasing journey and influence partners and stakeholders. This role leads surveys...
- Cloudflare is seeking an Indirect Tax Analyst to join our collaborative team. This position will have a broad range of responsibilities with a primary focus on Indirect Tax (US & International - VAT, GST, Sales Tax, Use Tax and Property Tax). This Indirect Tax Analyst will...
- ...thinking organization, apply now.We are currently seeking a Experience researcher + designer to join our team in Remote, Texas (US-TX), United... ...responsible innovation. We are one of the world's leading AI and digital infrastructure providers, with unmatched capabilities...Work at officeRemote workFlexible hours
NTT DATA
Austin, TX4 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Benchmark Researcher. Be the first to apply!
Related searches