Staff AI Evaluation Architect
Uncover
Uncover is building a central evaluation framework to measure model quality across teams. You will design reusable pipelines, benchmarks, and baselines, and build visualization tools to turn results into actionable insights for stakeholders. The role demands strong software engineering, experience with data/evaluation pipelines at scale, and the ability to translate research outputs into standardized metrics across domains. You’ll own the end-to-end delivery from requirements to deployment. #J-18808-Ljbffr Uncover
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Staff AI Evaluation Architect in San Francisco, CA vacancy
- Mercor is partnering with a leading AI research organization to engage experienced accountants for a project that evaluates how well AI systems perform real‑world accounting work. You will design task‑specific grading criteria and define what excellent work looks like....Suggested
- ...engineer to design tasks in materials science and engineering that test AI models performing expert work. You write realistic prompts, assemble a data room, and define grading criteria to make evaluation objective. You will run tasks against prompts, analyze results, and...Suggested
- Obsidian is seeking a part-time researcher to design evaluation challenges for frontier AI in drug discovery. You will build data rooms grounded in real sources, craft grading rubrics, and validate approaches to ensure defensible conclusions. The role emphasizes rigorous...SuggestedPart timeFlexible hours
- ...seeking an experienced CI/CD/Release Engineer in San Francisco to build and maintain the automated delivery pipelines for a fast-moving AI-enabled team. You will design cross-repo release orchestration, spin up ephemeral PR environments, and collaborate with QA and...Suggested
$190.3k - $261.5k
...achievement of company goals. About the Role The AI Solution Architect is a senior technical leader within Dolby's AI & Business Enablement... ..., and compute optimization at scale. Experience evaluating and integrating emerging AI technologies, frameworks, and...SuggestedFull timeLocal areaWorldwideFlexible hours- The Biological Computing Co. (TBC) is building next generation video generation models enabling robots to learn, plan, and act through imagined futures. You will set the technical direction for core research areas, translate breakthroughs into deployable systems, and collaborate...
$145.6k - $209.3k
...Identity & Access Management (IAM) Architect to help define, design, and... ...security objectives.• Evaluate emerging identity technologies... ...workforce insights, and people-first AI, our ability to reveal unseen... ...StatesTravel:Up to 25%Role:Sr. Staff IAM ArchitectDepartment:...$206.4k - $379.1k
...effortlessly produce impressive content. The AI Foundations team constructs a flexible,... ....We're looking for a Principal Architect to build and implement the AI framework for... ...layers, session analytics, and continuous evaluation frameworks.This role blends applied research...Full timeTemporary workLocal areaWorldwideFlexible hours$122k - $240.5k
Position Summary Google AI Architect/AI and EngineeringJoin our AI & Engineering team in transforming technology platforms, driving... ..., reliability, security, and cost.Design, fine-tune, evaluate, and govern LLM solutions with Gemini on Vertex AI (prompt/tool...Local areaVisa sponsorshipFlexible hours- ComfyUI is seeking an experienced engineer to architect testing and verification infrastructure for a daily shipping cadence. You will... ...perceptual canvas comparisons, and non-deterministic regression evaluation. This is an engineering role with heavy automation, CI, and an...Daily paid
- A leading AI-powered research company in San Francisco seeks a highly technical engineer to join a fast-growing team. You will be responsible for solving complex problems across the product design, while maintaining a high bar for quality and detailed execution. The role...
$164.7k - $266k
...lifecycle management (CLM).What you'll doWe are seeking a Lead AI Architect to turn enterprise data, metadata, relationships, and business... ...leaders, architects, engineers, analysts, and vendors to evaluate and implement strategic AI solutionsJob DesignationHybrid: Employee...Contract workWork at officeLocal areaRemote work2 days per week- Mercor partners with a leading AI research organization to engage experienced accountants. You will define what excellent accounting work looks like by designing task-specific grading criteria and scoring samples with rigorous written justifications. You will apply consistent...
- ...Draup is a Series A-funded agentic AI company building the intelligence layer for how... ...proprietary model development. Prototype and evaluate breakthrough AI capabilities — predictive... ...2 years in a principal engineer or lead architect role. ~ Demonstrated history of...Work at officeVisa sponsorshipFlexible hours3 days per week
$240k - $315k
...interpretable, and steerable AI systems. We want AI to be safe... ...executives and account architects to understand a customer’s security... ..., from discovery and initial evaluation through deployment,... ...policy: Currently, we expect all staff to be in one of our offices at...Work at officeVisa sponsorshipFlexible hoursAnthropic
San Francisco, CA5 days ago- ...role, you'll apply your expertise to help train next-generation AI systems. Your work will shape how models learn, reason, and perform... ...and operational challenges represented in Excel documents. Evaluate and simulate real-world business problems using advanced Excel features...Remote jobContract work
- Obsidian in San Francisco builds high-fidelity simulated work environments to evaluate AI agents in marketing contexts. You will design organic growth tasks, specify required documents and dashboards, and judge AI attempts against a standard rubric. We value hands-on experience...
- Mercor is seeking researchers to author AI evaluation tasks and original, executable problems for frontier models. You will source material, write prompts, and define grading criteria across subdomains with a focus on two areas in mathematics. Engagement is six weeks, part...Part timeImmediate start
- ...Deployed Software Engineer to lead the software layer beneath complex partner deployments, translating bespoke environment, data, and evaluation work into repeatable infrastructure. You will own architecture for deployment services, ensure reliability and observability at...
$142.6k - $261.5k
...to build a better working world. ServiceNow– ServiceNow AI Architect Manager In the digital economy, it takes more than good... .... Exercise judgment in selecting methods, techniques, and evaluation criteria for obtaining results. Provide direction and feedback...Summer holidayWorldwideFlexible hoursEY
San Francisco, CA18 hours ago- Mercor is seeking senior K-12 education professionals to build evaluation tasks for AI systems operating in large school district and public education contexts. The workflows are calibrated to the instructional complexity, stakeholder diversity, and student-outcome stakes...
- Obsidian is looking for an individual to create benchmark tasks evaluating AI models on financial document understanding. You will develop multi-step requests using real-world financial files, ensuring AI can follow instructions and generate structured responses. The role...
- Conductor is seeking memory subsystem architects to own the memory subsystem for a high-bandwidth system. You will design and implement... ...controllers, and interfaces with the XPUs/compute complex. You will evaluate LLm inference workloads, memory tiering implications, and...
- ...is the infrastructure for enterprises to build and orchestrate AI workforces. Our AI workers don't just communicate - they make decisions... ...to the processing of your personal data for the purpose of evaluating and selecting you as a candidate for the position. Your personal...WorldwideShift work
- Uber is seeking an AI Governance Lead within the Financial Risk Management (FRM) Advisory team in San Francisco. You will shape AI... ...initiatives. The role requires building scalable control architectures, evaluating AI use cases for ICFR risk, and coordinating with CAO, finance,...
- ...SFO, CA Duration: Long term contract Description As an Enterprise Architect, you will be responsible for managing Enterprise‑wide solutions... ...specific expertise in Digital, E‑Commerce, Data, MarTech, AI/ML and Cloud solutions. Lead the Architecture and design of business...Long term contract
GSPANN Technologies
San Francisco, CA18 hours ago $125k - $250k
...Job Description Job Description AI Partner Ecosystem Acceleration & Strategic Alliance Architect Company: HireNow Staffing (Direct Placement Partner) HireNow... ...Identify emerging AI technology partners and evaluate strategic market opportunities. Build executive...Full timeImmediate startRemote workVisa sponsorshipHireNow Staffing
San Francisco, CA11 days ago- ...executing full-cycle recruiting for a range of roles including AI/ML research and engineering. Successful candidates will design interviews... ...relationships within the AI community, and ensure rigorous evaluation processes to maintain high hiring standards. #J-18808-Ljbffr...
- Senior AI Architect - Multi-Agent Systems & Platform Infrastructure Senior AI Architect - Multi-Agent Systems & Platform Infrastructure... ...and Semantic Kernel components. Develop and refine test plans, evaluation pipelines, and debug tools for cross-agent messaging and...Full timeWork at officeRemote work
$180k - $240k
...Senior $180k - $240k About the Role We are seeking a Senior AI Agent Architect to design and deploy autonomous, multi-agent systems using... ...LLMs with external enterprise databases and APIs Define evaluations, safety guardrails, and reinforcement loops for agent tasks...Full time
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Staff AI Evaluation Architect. Be the first to apply!
