AI Evaluation Lead: Real-World Systems Benchmarking
SupportFinity
A cutting-edge AI technology firm in San Francisco is seeking an Evaluation Lead to drive the assessment of AI model performance. You will design evaluation methodologies, automate evaluation processes, and oversee various evaluation strategies. The ideal candidate has extensive experience in AI model evaluation and is proficient in Python. This high-impact role demands strong collaboration skills and a startup-ready mindset to thrive in a fast-paced environment. #J-18808-Ljbffr SupportFinity
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the AI Evaluation Lead: Real-World Systems Benchmarking in San Francisco, CA vacancy
- DoorDash is building an AI Research org from the ground up. We seek founding researchers... ...operations to translate research into real-world impact and publication at top venues.... ...-scale experiments, including pre-training, RL, and evaluation #J-18808-Ljbffr DoordashusaSuggested
$130k - $155k
Lead, Frontier AI Systems, Centre for AI Excellence page is loaded## Lead, Frontier... ...authorization support.The World Economic Forum, committed... ...their responsible adoption and real-world impact through a... ...system-centric frameworks for evaluation, assurance, and deployment,...SuggestedRelocation packageShift work3 days per week- ...focused on building next-generation conversational AI interfaces. The ideal candidate will have deep... ...ownership of end-to-end development of agentic systems and collaboration with cross-functional teams to create scalable real-time interfaces. Join a well-funded startup at...Suggested
$117.2k - $313.7k
...SalesforceSalesforce is the #1 AI CRM, where humans... ...a way of life. The world of work as we know... ...at the company leading workforce... ...Salesforce.Distributed Systems Software Engineer -... ...proficiency with solving real-world data... ...reliably.Critically evaluate code (Human or AI-generated...SuggestedFull time- ...relationships. Some of the world's most forward-... ...workflows with AI. We're a small, tight... ...8 AM PT) so we get real-time overlap with... ...ground up. Plain is leading the charge in... ...guiding technical evaluations, and leading POCs that... ...webhooks, and how systems work under the hood...SuggestedWork at office3 days per week
$117.2k - $313.7k
...Salesforce is the #1 AI CRM, where humans... ...a way of life. The world of work as we know... ...career at the company leading workforce... ...Salesforce. Distributed Systems Software Engineer -... ...proficiency with solving real-world data... ...reliably. Critically evaluate code (Human or AI-...Salesforce.Com Inc
San Francisco, CA21 hours ago- ...agents that accelerate the world's largest enterprise... ...autonomous development systems — shipping code, reviewing... ...Integrators Partnerships Lead at Factory, you'll own a... ...written. You'll have real influence over how we engage... ...deliver transformative AI-powered development...
- Vals AI in San Francisco is seeking exceptional... ...generation of AI benchmarks. You will create high-impact, challenging evaluations that push the boundaries... ...influence how the world evaluates AI systems. We are building the... ...ability of LLMs to perform real-world #J-18808-...
$200k
...Senior Security Lead - design and... ...implementation of systems that protect... ...multi‑million user AI platform... ...future of how the world measures artificial... ...in securing real‑time, interactive... ..., this open evaluation platform has become... ...crowdsourced benchmarks and human preference...Work experience placementVisa sponsorship$181k - $285k
...on a journey to build the world’s most advanced AI infrastructure ecosystem.... ...the RoleOpenAI is seeking a Real Estate Lead to own land acquisition... ...economics.Proactively source and evaluate land parcels at scale to... ...of the capabilities of AI systems and seek to safely deploy...Work at officeLocal areaRemote workFlexible hoursShift work$190k - $230k
...how we shape a better world.We collaborate with public... ..., design, delivery, systems integration, assurance,... ...Systems Design Coordination Lead based in our San... ...for someone who has a real passion to learn, a high... ...have applied you will be evaluated and potentially moved on...Contract workFor contractorsWork at officeRemote workFlexible hours$141.2k - $278.3k
Position Summary Google AI Lead Architect/AI & Engineering:... ...and cost.Design, fine-tune, evaluate, and govern LLM solutions with... ...security and governance for AI/ML systems (data privacy, model... ...solutions in a professional, real-world environment.Strong understanding...Local areaVisa sponsorshipFlexible hours- ...unlock liquidity for the world. Backed by leading investors like PayPal... ...toward a future where AI is embedded into how... ...engineering translating real operational pain... ...company, building the systems, standards, and internal... ...Standardize how we build, evaluate, and scale AI...Work at officeRemote work2 days per week
MESH
San Francisco, CA4 days ago $186.1k - $300.55k
...disconnected from business systems of record, costing... ...What you'll doAs a Lead Product Manager for Agentic AI Platform, you will... ...and AI Agent evaluation within the Intelligent... ...to measure, benchmark, and improve agentic... ...trust and making the world more agreeable for...Contract workWork at officeLocal areaRemote workShift work2 days per week- Mercor is partnering with a leading AI research organization to engage experienced sales engineers for a project focused on evaluating how well AI systems perform real-world technical sales work. You will define what excellent work looks like: designing task-specific grading...
- About Archetype AI Archetype AI is developing the world's first AI platform to bring AI into the real world. Formed by an exceptionally high... ...AI is seeking a hands‑on Evaluation Lead to build and assess model... ...tracking competitive industry benchmarks. This is a high‑impact...
$167.3k - $261.4k
Senior Research Lead - AI Security Portfolio page is loaded## Senior... ...on securing advanced AI systems, understanding their cyber capabilities... ...research, cyber capability evaluation, and infrastructure... ...cutting-edge research with real-world policy impact.**Qualifications...Fixed term contractWork experience placementRemote workWork from home- ...6-00093 Responsibilities: Lead the data quality evaluation by investigating all clinical... ...potential impact on study systems setup, study conduction,... ...approach Knowledge of Real-World data sources and processes... ...Artificial Intelligence (AI) Project Management skills...For contractors
- Vals AI, Inc. in San Francisco is seeking strong engineers to own the leaderboards that evaluate real-world tasks by LLMs. You will test and benchmark new models across domains like law, tax, coding, finance and more, analyzing error modes and collaborating with the communications...
- ...seeking a Member of Technical Staff to lead its new robotics vertical. This role entails defining benchmarking methodologies and producing leaderboards to evaluate robotics capabilities. Ideal... ...will possess a strong interest in AI and robotics, alongside robust coding...
- ...enterprise deployments. This role requires deep expertise in real‑time voice systems and experience with major contact‑center platforms. You will... ..., and shape a product that enables scalable, enterprise‑grade AI voice agents as part of Maven's #J-18808-Ljbffr Maven AGI
- ...seeking a Senior GNC Engineer to take responsibility for GNC algorithms and systems that control their rockets. You will design and implement algorithms, build simulations in Simulink, and support real launches. Ideal candidates have over 5 years of aerospace GNC experience...
$200k - $250k
About Distyl AI Distyl is an applied AI technology... ...with the world’s most ambitious institutions... ...self-constructing systems, the development of... ...is backed by leading investors including... ...systems deliver real business value.This... ...agents, retrieval, evaluation, and execution into...Work at office3 days per week$200k - $250k
About Distyl AI Distyl is an applied AI technology... ...with the world’s most ambitious institutions... ...self-constructing systems, the development of... ...is backed by leading investors including... ..., code quality, evaluations, integrations, and... ...modern AI tools, and real-world business problemsOwnership...Work at office3 days per week- ...across a business—until generative AI. Today, AI is the number one driver... ...help clients shift to the New using leading-edge technologies on some of the coolest... ...value, and reliability of deployed systems.The Work:• Formulate real-world problems into practical, efficient,...Full timeWork experience placementLive inWork at officeLocal areaShift work
$185k - $220k
...AreNotion is the collaborative AI workspace where teams... ...building a business’s system of record to making and... ...and technically fluent Lead, IT Audit to join our... ..., testing, deficiency evaluation, remediation, and... ...excited to use AI as a real collaborator in their work...Local area$155.7k - $186k
Fin is the AI Customer Agent company on a mission... ...help desk for one single system that is designed to... ...software companies in the world with nearly 30,000... ...our brand.As Brand Voice Lead, Systems, you’ll help make... ...workflows, editorial evaluations, or structured guidance...Work at officeLocal areaWork from homeFlexible hours3 days per week$175k - $225k
...clients in communities around the world.Taskrabbit is a hybrid... ...looking for a hands-on builder to lead our ‘AI for Work’ efforts. This role... ..., error rate) to validate real impact, not just adoptionEnable... ...we serve, breaking down systemic barriers, and transforming lives...Work at officeImmediate startFlexible hours2 days per week- ...300 products for the world's leading startups, enterprises,... ...that engine at agentic AI, and we are looking for... ...them to make it real. As an AI Strategist... ...specifically agentic systems, create real business... ...technologies, enabling you to evaluate technical feasibility...Remote work
$357k
...applications, processes, and AI into a single,... ...process to power real‑time orchestration... ...Responsibilities Lead AI Research Scientist... ...AI with real‑world impact for developers... ...workflow graphs, agent evaluation frameworks, and... ...within large‑scale systems. Strong publication...Work at officeFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Evaluation Lead: Real-World Systems Benchmarking. Be the first to apply!
