LLM Benchmark Engineer — Lead Leaderboard Insights
Vals AI, Inc.
Vals AI, Inc. in San Francisco is seeking strong engineers to own the leaderboards that evaluate real-world tasks by LLMs. You will test and benchmark new models across domains like law, tax, coding, finance and more, analyzing error modes and collaborating with the communications team to publish results. The role involves working with major labs and institutions, building model library integrations, and maintaining benchmarks infrastructure in a fast, sprint-driven environment. #J-18808-Ljbffr Vals AI, Inc.
Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the LLM Benchmark Engineer — Lead Leaderboard Insights in San Francisco, CA vacancy
- Vals AI in San Francisco is seeking engineers to own leaderboards that evaluate LLMs across tasks including law, tax, coding, finance, and more. You will test and benchmark new models as released, analyze error modes, and work with our communications team to publish results...Suggested
- Dynamo AI is seeking a candidate to lead LLM evaluation and benchmarking in San Francisco, California. You will generate high-quality data and develop innovative methods for assessing the safety and helpfulness of LLMs. The role requires domain knowledge in evaluation...Suggested
- Dynamo AI in San Francisco seeks an ML Engineer focused on evaluating LLMs, benchmarking, and ensuring safe, responsible AI in production. You will own end-to-end evaluation pipelines and contribute to synthetic data generation. You will collaborate with researchers to...Suggested
- ...technical leader to head its robotics coverage, developing benchmarking methodologies, leaderboards, and datasets that set industry standards. You will... ...to translate complex robotics concepts into actionable insights. This role blends product, research, and client-facing...Suggested
- Vals AI in San Francisco is seeking exceptional researchers and research engineers to design and build the next generation of AI benchmarks. You will create high-impact, challenging evaluations that push the boundaries of what we can measure in foundation models. This...Suggested
- ...you'll be the one who decides what those insights are. You'll sit with our top customers,... ..., revenue deep dives, cross-customer benchmarks. Find the insights that matter. Clean... ...externally. Partner closely with Product and Engineering throughout your role. What we're...For contractorsLocal areaDay shift
- Artificial Analysis, Inc. in San Francisco is seeking a Member of Technical Staff to lead its new robotics vertical. This role entails defining benchmarking methodologies and producing leaderboards to evaluate robotics capabilities. Ideal candidates will possess a strong...
$118k - $147.5k
...efficiently. Within Marketplace, the Market Insights team is responsible for the health and... ...with Data Science, Product, Engineering, and other teams to deliver business, product... ...strategically and tackle open-ended problems, leading to analysis, recommendations, and next...Hourly payWork at officeLocal area3 days per week$170k - $245k
...250+ million raised to date.About the roleAs a Distributed LLM Inference Engineer, you will help systems and optimizations that push the boundaries... ...role to Anyscale as it allows us to achieve a market leading position for AI infrastructure.As part of this role, you willIterate...Work at office$160k - $230k
...anInference Frameworks and Optimization Engineer to design, develop, and optimize distributed... ...opportunity to shape the future of LLM inference infrastructure, ensuring scalable... ...algorithms, and models. We have contributed to leading open-source research, models, and...Full time$98k - $140k
...work with product and engineering teams to build systems... ...launch new models with leading research labs —... ..., Google, and others. Benchmark across dimensions: quality... ...“jagged frontier” of LLM capabilities and how AI... ...— You can self-serve insights from large datasets, whether...Live inLocal area$150k - $230k
...solutions for the consulting, architecture, engineering, project controls, procurement,... ...innovative Mechanical Engineering Group Lead to join our growing Emeryville, CA office... ...client-side experience, and supplier-side insight to every project. We infuse our curiosity...For contractorsWork at officeLocal area$187.3k - $237.2k
...Supply Chain, on the central CustomerQuality Engineering team. You’ll join a global group of... .... They are then responsible for driving insights back to the internal engineering teams to... ...our most strategic customers. You will lead a team that delivers a “white glove” experience...Full timeTemporary workLocal areaFlexible hours$180k - $270k
...and collaborates.We’re assembling a diverse, world-class team—engineers, designers, researchers, and product minds—focused on creating... ...develop next-generation wearable and AI-connected products. As a Lead RF Systems & Desense Engineer, you will own the RF system architecture...Full timeTemporary workLocal areaFlexible hours$180.6k - $289.3k
...Visa seeks a Senior Research Manager for its North America Insights & Analytics team in San Francisco. This hybrid role will focus on campaign measurement and effectiveness, evaluating marketing initiatives for clients. The candidate should have substantial experience...Visa
San Francisco, CA12 hours ago$127k - $269k
Figma is seeking a strategic, data-driven Voice of the Customer (VOC) Program Manager to lead a company-wide VOC program. This role is pivotal in surfacing customer insights from various sources and ensuring they are acted upon to improve processes and products. The ideal...Full timeRemote work- ...Technical Success, Product, Engineering, RevOps, and Strategic Finance... ...customer feedback into rigorous insight products that help leaders... ...Applied Data Science & Insights Lead for GTM Intelligence... ...Develop customer segmentation and benchmarking frameworks across products, industries...Work at officeRelocation package
- Okta is seeking a Communications Data and Insights Manager to build the analytics engine informing our corporate communications strategy. You will translate data into actionable insights, measure campaign performance, identify trends, and set KPIs that shape a modern,...
- ...opportunity to affect product launch support, forecast planning, and executive decision support. You will work with claims, EHR, and other healthcare data, applying SQL and Python to generate actionable insights that inform commercial strategy. #J-18808-Ljbffr ProcDNA
- ...is seeking a Senior Web Analyst for its eCommerce Conversion team. This role involves leading web analytics reporting and conducting in-depth analysis for actionable business insights. You'll collaborate with brand leads and manage performance reports to improve conversion...
- Pacira BioSciences, Inc. is seeking a senior Associate Director of Advanced Analytics to drive insights from commercial data sources and build scalable analytics capabilities across marketing, sales, finance, and managed care. You will partner with Insights, Marketing,...
- ...real‑time multimodal LLM for real life, transforming... ...data into valuable insights and knowledge that... ...a hands‑on Evaluation Lead to build and assess model... ...closely with research and engineering teams to develop... ...competitive industry benchmarks. This is a high‑impact...
- Exa is hiring a Revenue Insights Manager to own the numbers that run GTM. You will build the analytics layer from the ground up, define... ...dashboards for CROs and founders. You will partner with Data Engineering on pipelines and manage Salesforce, Clickhouse, Hubspot and...
- ...in California, is seeking a Convergent TV Planner, Strategy & Insights. The role focuses on shaping media strategy based on data, audience... ...responsibilities include developing audience-first strategies, leading RFP responses, and crafting compelling narratives. Candidates...
- ...Search Intelligence Manager in San Francisco to analyze SEO and AEO performance data. You will transform search data into actionable insights that guide our digital strategies. The role requires 3-5 years of experience in performance marketing or digital analytics, with...
- Linktree is seeking a Content & Audience Insights Manager in a hybrid role across Los Angeles and San Francisco. The role focuses on turning audience signals, content performance, and proprietary data into sharper content ideas and business narratives. You will own the...
- A leading consulting firm is seeking a Senior Consultant specializing in Life Sciences and Business Insights. The role involves analyzing data to provide actionable insights, managing client interactions, and leading transformation projects. Candidates should possess significant...
- ...internet’s largest sources of information. For more information, visit We are looking for a Marketing Science Partnerships & Insights Senior Lead to serve as a strategic driver of our measurement ecosystem. This role requires deep expertise in managing high-stakes...Contract workShift work
- ...hands-on analytics leader to build customer-facing dashboards and insights for trades businesses. You will own the core BI views, clean... ...across Looker, Tableau, and Power BI, and partner with Product and Engineering to scale analytics. #J-18808-Ljbffr Broccoli AI
- Exa is looking for a Revenue Insights Manager to own the numbers that drive our GTM engine. You will build the analytics and insights layer from the ground up, defining... ...own data-driven cadences across the organization, lead weekly pipeline reviews, forecast calls, and board...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to LLM Benchmark Engineer — Lead Leaderboard Insights. Be the first to apply!
