Senior AI Systems Engineer, Evaluation & Orchestration
Servicenow
ServiceNow seeks a senior backend/infrastructure engineer to build and operate evaluation and observability tooling for its enterprise AI agent platform. You will design runtimes for multi-turn agent scenarios, implement calibration against human labels, and drive scalable, reproducible experiments across production-scale datasets.
The role emphasizes Python/Go expertise, OpenTelemetry tracing, and building non-deterministic systems that yield meaningful evaluation signals.
#J-18808-LjbffrVacancy posted 3 hours ago
Similar jobs that could be interesting for youBased on the Senior AI Systems Engineer, Evaluation & Orchestration in Mountain View, CA vacancy
- ...MetAntz, a fast-moving AI startup in Palo Alto, seeks experienced software engineers who thrive in ambiguity and want to own end-to-end services from design to production... ...frontends, backends, and cloud-native distributed systems, delivering reliable APIs and scalable...Senior
MetAntz
Palo Alto, CA3 hours ago - ...Jobtailor is seeking a senior backend architect to architect and scale orchestration engines for complex AI workflows with low latency. You will design high‑performance RAG infrastructure, integrate vector databases, implement automated simulations, and productionize...Senior
Jobtailor
Menlo Park, CA3 hours ago - ...Affirm is building its own bank and this Senior Software Engineer role sits in the engineering team that runs it. You’ll ship core banking systems, data platform, and AI-enabled tooling on a small, autonomous squad, with exposure to bank leadership and a path to impact...SeniorRemote work
Affirm
Palo Alto, CA3 hours ago - ...Intuit is seeking a Senior Staff Software Engineer in Mountain View, California. This role involves driving significant technology... ...should have over 10 years of experience in systems/software development and expertise in AI technologies. Intuit offers a competitive...Senior
Intuit
Mountain View, CA3 hours ago - ...A leading tech company is seeking a Senior Software Engineer for AI and Infrastructure. The ideal candidate will possess strong programming expertise in C++, Java, or Python, with a focus on software design and architecture. Responsibilities include writing and testing...SeniorFull time
Google
Sunnyvale, CA3 hours ago - ...CoreWeave is looking for a Senior Software Engineer II to enhance our Kubernetes-native platform for AI workloads. You will design and manage Kubernetes services while mentoring junior engineers. Ideal candidates will have 5-8 years of experience in software engineering...SeniorRemote workFlexible hours
CoreWeave
Sunnyvale, CA11 hours ago - ...A leading technology company in Sunnyvale, CA is seeking a Software Engineer III to develop next-generation technologies related to AI and ML. This role requires a Bachelor's degree and experience in programming, particularly with Python or C++, along with expertise in...Senior
Google Inc.
Sunnyvale, CA3 hours ago $152k - $241.5k
...models are foundational to American AI leadership and cybersecurity, and that... ...scrutiny. Our AI Safety & Security Engineering team builds and evaluates AI-powered tooling that helps find,... ...We are looking for an Evaluation/ML-Systems Engineer to own how we measure the program...Full timeRemote work$178k - $321k
...an early but working AI-native capability: a multi... ...agentic runtime and evaluation harnesses that make... ...This is a two-person engineering team: you deploy, debug... ...build production-grade systems end to end on what exists... ...runtime and harness: orchestration, multi-model routing...Senior- ...NVIDIA in Santa Clara, CA is seeking a Senior Software Engineer for Agent Simulation & Evaluation to advance AI workloads across GPUs and LPUs. You will build agentic systems, develop evaluation tools, and collaborate with inference, hardware, and product teams. Required...Senior
NVIDIA
Santa Clara, CA3 hours ago - ...NVIDIA in Santa Clara, CA, is seeking a Senior Software Engineer for Agent Simulation and Evaluation in Finance. You will design AI-native systems to automate simulation, evaluation, and reporting across GPU and LPU platforms. Responsibilities include building experiments...Senior
Nvidia Corporation in
Santa Clara, CA3 hours ago - ...NVIDIA is seeking a Senior Software Engineer for Agent Simulation & Evaluation to help build agentic systems and provide performance analytics across GPU and LPU platforms. The role emphasizes Python tooling, AI agents, and collaboration with inference, hardware, runtime...Senior
NVIDIA Gruppe
Santa Clara, CA3 hours ago - ...leader in enterprise orchestration, Workato helps businesses... ...and experiences. Its AI-powered platform... ...exceptional Search/AI Engineer with experience in Search... ...of intelligent search systems that leverage machine... ...Own the design of evaluation frameworks for offline...Full timeRemote workFlexible hours
Workato
Palo Alto, CA22 hours ago - ...Moveworks is seeking a Senior Software Engineer to own the judgment layer of the Agent Eval Platform. You will design scoring, calibration... ...trajectories in enterprise environments. You will build orchestration at scale, enhance observability with OpenTelemetry, and...Senior
Moveworks
Mountain View, CA3 hours ago $200k - $400k
...machines at scale. At Scout AI, we’re developing Fury,... ...We're looking for a Senior or Staff AI Engineer to join the Fury Orchestration Team with a deep... ...directly influence how our system reasons, plans, remembers... ...: model training and evaluation, reinforcement learning...SeniorFull timeRelocation packageScout Ai
Sunnyvale, CA22 hours ago$160k - $198k
...and celebrates all of our team members.What You’ll DoAs a Senior AI Systems Engineer, you will architect, deploy, and manage the critical... ...raw data and high-performance AI models.Infrastructure & Orchestration: Deploy, scale, and manage resilient infrastructure services...SeniorLocal area- About UsHippocratic AI is the leading generative AI company in healthcare. We have the only system that can have safe, autonomous, clinical... ....About the RoleAs an AI Engineer at Hippocratic AI, you’ll play... ...improvement in model evaluation, safety testing, and observability...SeniorWork at office
$144k - $236k
...needs of the team. The Video AI team sits at the heart of... ...intelligent content recommendation systems, positioning our company as... ...one billion members. As a Senior AI Software Engineer, you will own end-to-end... ...-art models, designing and evaluating online experiments, to...SeniorFor contractorsWork at officeFlexible hours$190.9k - $334.1k
...DescriptionIt all started when engineer Fred Luddy wrote code... ..., ServiceNow is the AI control tower for... ...-scale search systems that power Now Assist,... ...multi-agent systems — orchestration, tool use, planning loops... ...search, re-ranking, and evaluation — as a critical dependency...SeniorWork experience placementWork at officeImmediate startRemote workFlexible hours$139k - $257.55k
...consistency, reusable design systems, and collaboration... ...next generation of AI-native creative systems... ...for forward-thinking engineers who are excited to explore... ...experience.Develop evaluation and quality frameworks... ..., structured outputs, orchestration frameworks, multimodal...SeniorFull timeTemporary workLocal areaWorldwide$227.5k - $300k
...the transformation to AI-enabled software-... ...We are seeking a Senior Staff AI Engineer with a combination of... ...scalability of the resulting system. Responsibilities:... ...of agentic orchestration by designing and implementing... ...engineering, and model evaluation techniques. ~...SeniorFull timeWork at officeWorldwideFlexible hoursShift workSonatus
Sunnyvale, CA22 hours ago$185k - $215k
...this role, you will operate as a Staff AI Systems Engineer, setting technical direction across... ...guide the design, implementation, and evaluation of our Advanced Driver Assistance... ...distributed systems, cloud infrastructure, or orchestration tools (e.g., AWS, Azure, GCP, Airflow...Odd jobWork experience placementInternshipLocal areaWorldwide$163.2k - $220.8k
...growth. Wilson Sonsini is looking for a Senior AI Security Engineer to join the Security Operations team.... ...law firms builds and deploys AI systems with security and trust at their core... ...function: designing secure AI pipelines, evaluating MCP server integrations and agentic...SeniorFull timeWork experience placementRemote workWorldwideShift work$168k - $264.5k
...architecture, silicon, systems, and manufacturing. The... ...performance loss.We're hiring a Senior Manufacturing & System Co-Design Workflow Engineer to lead the methodology... ...-loop checkpoints, and evaluation-based CI gates running... ...that makes AI genuinely usable in a rigorous...SeniorFull timeImmediate start$191k - $315k
...LinkedIn’s Enterprise AI team creates... ...as-a-judge, prompt engineering, embedding-based retrieval... ...and agentic AI systems that perform... ...answer questions, and orchestrate multi-step, tool-... ...environments.Evaluate and drive key engineering... ...experience at the Senior Staff / L7 /...SeniorFor contractorsWork at officeFlexible hours$226.15k - $252k
...audio logic within the orchestration framework.Guarantee... ...production by triaging system issues, debugging code... ...in Computer Science, Engineering, Computer Information... ...for model deployment, evaluation, and optimization; Root... ..., we are a pioneering AI lab with exceptional interdisciplinary...SeniorFull timeTemporary workWork at office- ...NVIDIA is seeking a Senior Agentic AI Software Engineer (Finance) to build agentic systems and advance AI inference workloads in real-world finance contexts. You will... ...at scale. This role collaborates with evaluation teams, researchers, and engineers to ensure high...Senior
Nvidia Corporation in
Santa Clara, CA3 hours ago - ...NVIDIA in Santa Clara, CA is seeking outstanding AI systems engineers to advance the inference software stack. You will design and optimize kernels, build new abstractions for LLM serving engines, and contribute to accelerators and runtimes that power large language models...Senior
NVIDIA
Santa Clara, CA3 hours ago - ...We are seeking an Agent Engineer to help build the next generation of AI-powered workflows and intelligent systems. In this role, you will design... ...and planning, including orchestration and multi-step workflows.... ...and operationalize evaluation frameworks focusing on accuracy...Senior
Techvilla Solutions
Los Altos, CA2 days ago $129.4k - $198.4k
.... About the Organization:The Evaluation team builds and evolves the evaluation... ...with Autonomy, Simulation, Systems, and Safety teams, we act as... ...interpretable insights to engineering teams and leadership,... ...testing best practices. Leverage AI-assisted development tools and...SeniorFull timeLocal areaRemote workWork from homeFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior AI Systems Engineer, Evaluation & Orchestration. Be the first to apply!
Related searches
- ai ml engineer Mountain View, CA
- machine learning ai engineer Mountain View, CA
- ai developer Mountain View, CA
- ai engineer Mountain View, CA
- ai prompt engineer Mountain View, CA
- senior ai engineer Mountain View, CA
- senior windows systems engineer Mountain View, CA
- software system engineer Mountain View, CA
- mission system engineer Mountain View, CA
- healthcare systems engineer Mountain View, CA