LLM Agent Evaluation Engineer — Build Intelligent Agents
Bytedance
ByteDance Applied Machine Learning Ark team in San Jose develops and operates LLM service platforms that offer MaaS solutions to businesses and researchers, spanning model serving, tool calling, and analytics.
This full-time role emphasizes designing evaluation systems for LLM agents, building benchmarks and pipelines, analyzing traces and feedback, and collaborating with research, platform, and product teams to bring methods into production.
#J-18808-LjbffrVacancy posted 8 hours ago
Similar jobs that could be interesting for youBased on the LLM Agent Evaluation Engineer — Build Intelligent Agents in San Jose, CA vacancy
- ...The Apple Intelligence Platform Experience Validation team builds the tooling and automation that keeps Apple Intelligence... ...of automated model evaluation: standing up LLM-as-a-judge in existing and new... ...into it quickly.Strong software engineering fundamentals — able to define...Suggested
Socket.dev
Cupertino, CA8 hours ago - ...that span text and multimodal capabilities. You will help build end-to-end MaaS solutions, data pipelines, and evaluation systems across US and international markets. The internship blends system engineering with cutting-edge AI research, offering hands-on experience...SuggestedInternship
Pangle
San Jose, CA8 hours ago $184k - $287.5k
Artificial intelligence is shifting from passive help to autonomous... .... We seek a Senior Engineer to lead technical... ...deploying advanced AI agent frameworks and local runtimes... ...environments. You will build the technical roadmap... ...understanding of LLM inference pipelines (Ollama...SuggestedFull timeLocal areaShift work- ...ServiceNow in Santa Clara, CA is seeking a leader for the Build Agent evaluation framework. You will own eval strategy, roadmap, telemetry, and cross-team quality standards, guiding an 8‑engineer team to deliver scalable, data‑driven improvements. You will benchmark...Suggested
ServiceNow
Santa Clara, CA8 hours ago $128k - $256k
...team combines system engineering and machine learning... ...Large Language Model (LLM) service platforms that... ...China. We are building full-stack, end-to-end... ...model alignment, and intelligent agent systems. Beyond model... ...the year. Design evaluation systems for LLM-based...SuggestedTemporary workInternshipLocal areaBytedance
San Jose, CA8 hours ago- ...Otter.ai is seeking a Senior AI Agent & LLM Engineer in Mountain View, CA. You will build and improve AI systems behind Otter's conversational knowledge engine... ...AI Chat, spanning user-facing agent experiences, evaluation systems, and the shared platform for scalable, reliable...
Otter.ai
Mountain View, CA8 hours ago - ...A forward-thinking technology firm in San Jose is looking for an LLM Engineer to design and deploy large language models that drive innovative AI solutions. The role includes designing models, building inference pipelines, and collaborating with cross-functional teams...
Take2 Consulting, LLC
San Jose, CA8 hours ago $171.6k - $222.2k
...world-class scientists and engineers to pioneer the next... .... Join the Amazon Kiro LLM-Training team and help... ...models specialized in code intelligence- Invent and implement... ...teamThe AWS Developer Agents and Experiences (DAE) team... ...time to focus on a build. We also value in-person...Work at officeLocal areaWorldwideFlexible hours- ...Nutanix is seeking a software engineer based in Santa Clara, California, to develop AI-powered product features. The role requires building backend services and APIs, implementing LLM workflows, and collaborating with product managers and designers. The ideal candidate...
Nutanix
Santa Clara, CA8 hours ago $155k - $185k
...are looking for a Senior AI Agent & LLM Engineer who combines strong... ...AI quality. You will help build and improve the AI systems... ...-facing agent experiences, evaluation systems, and the shared platform... ...collaboration. Using artificial intelligence, Otter generates real-time...Permanent employment- ...ServiceNow is seeking a Software Engineering Manager for Build Agent to lead an 8-engineer team focused on evaluation infrastructure, model benchmarking, and telemetry. The role drives strategy, cross-team collaboration, and scalable processes to ensure robust evaluation...
ServiceNow
Santa Clara, CA8 hours ago $139k - $257.55k
...Collaboration, focuses on building foundational... ...create together intelligently and efficiently.... ...users and AI agents. Our mission is to... ...forward-thinking engineers who are excited to... ...Design and integrate LLM-powered and... ...experience.Develop evaluation and quality frameworks...Full timeTemporary workLocal areaWorldwide$166.5k - $291.4k
...DescriptionIt all started when engineer Fred Luddy wrote code... ...inspired Fred to build a company that could do... ...DescriptionThe teamBuild Agent is ServiceNow's AI... ...responsible for the Build Agent evaluation framework, model... ...evaluation systems for LLM-based products — scoring...Work at officeImmediate startRemote workFlexible hours$185k - $230k
...Senior Software Engineer, AI Agent & LLM Mountain View, CA The Opportunity... ...AI quality. You will help build and improve the AI systems... ...facing agent experiences, evaluation systems, and the shared... .... Using artificial intelligence, Otter generates real-time...Permanent employmentContract workFor contractorsFor subcontractorWork at officeOtter.ai
Mountain View, CA8 hours ago- NVIDIA is seeking highly motivated Engineers in Santa Clara, CA, to enhance AI capabilities... ...tasks. You will develop and integrate AI agents to improve productivity, collaborating closely... ...demands strong proficiency in advanced LLM technologies and the ability to work...
- ...is offering a Machine Learning Engineer Intern position for the 2027... ...and large language models to build scalable AI systems. You will explore Enterprise AI Agent technologies, advanced dialogue... ...and AI call capabilities, and intelligent decision‑making with practical...Summer workInternship
TikTok
San Jose, CA8 hours ago - ...TikTok's Commercial AI-CRM and Transaction team focuses on advertiser growth algorithms, leveraging LLMs and Agent tech to build intelligent systems for business objectives, growth, and decision‑making. You will contribute to enterprise‑grade AI initiatives, exploring...Internship
TikTok
San Jose, CA8 hours ago - ...With a strong focus on intelligent mobility, XPENG is... ...connectivity. Our team builds platform capabilities... ...Machine Learning Engineers to improve efficiency... ...We are building an LLM‑powered agent to help MLEs collect... ...documents, metrics, and evaluation results. Use LLM‑based...Internship
$152k - $241.5k
We are now looking for a Senior Software Engineer for Agent Simulation & Evaluation! Today, NVIDIA is tapping into the unlimited potential of AI to define... ...they map to future hardware.What you will be doing:Building agentic systems that automate simulation, evaluation,...Full time$244.14k - $413.16k
...a strong focus on intelligent mobility, XPENG is... ...exceptional Research Engineers / Scientists to... ...systems that allow agents to plan over long... ...data. You will help build the learning... ...learning methods for LLM-driven agents and... ...infrastructure platform.Evaluation and benchmarking...Full time$138k - $206k
...selves. Together, we’re building a better tomorrow for... ...(Artificial General Intelligence) Computing Lab is dedicated... ...hardware and software engineers to identify and... ...for modern and emerging LLM workloads.We are seeking... ...resources, and evaluate how emerging workloads...Work experience placementWork at officeFlexible hours$184k - $287.5k
...seeking an AI Compiler Engineer with deep expertise... ...role is focused on building AI-driven compiler intelligence for production... ...efficiency, and advancing LLM-enabled workflows... ...experimentation, evaluation, and deployment... ...LLM-based systems, agents, or developer tooling...Full timeRemote work$184k - $287.5k
...We are now looking for a Senior High-Performance LLM Training Engineer!NVIDIA is seeking experienced engineers specializing in performance analysis... ...deep learning platform stack, from drivers to DL frameworks.Build and support NVIDIA submissions to the MLPerf Training...Full timeWork experience placement$272k - $431.25k
...NVIDIA is seeking a Principal Engineer to drive the performance of large-scale AI training... ...will analyze and optimize frontier-scale LLM workloads running on thousands of GPUs, drive... ...across NVIDIA’s AI software stack.Build and refine performance models, workload characterizations...Full time$184.7k - $324.8k
...Learning Security (Agent Systems), SEAR... ...right. Our Security Engineering & Architecture (... ...users worldwide, building security into... ...responsible for developing intelligent security systems... ...adversarial evaluations that reflect... ...researching or securing LLM-based or tool-...WorldwideRelocation- ...motivated AI / Machine Learning Engineer with hands-on experience in Intelligent Systems and Generative AI to join... ..., or text/image generation Build and optimize data pipelines for... ...and product teams Perform model evaluation, tuning, and performance optimization...Full timeInternshipRelocation
Hudson Manpower
San Jose, CA1 day ago - ...NVIDIA is seeking a Senior Software Engineer for Agent Architecture and Evaluation to build agentic AI systems and analyze inference workloads in realistic settings. You will work with teams advancing evaluation across software and hardware to improve accuracy and efficiency...
NVIDIA Gruppe
Santa Clara, CA8 hours ago $150k - $180k
...ZEDEDA Inc. in San Jose, CA is seeking a Senior Software Engineer to design AI agents and enhance complex systems. Candidates must have significant experience in building and operating microservices and distributed architectures. This role offers a competitive base pay...ZEDEDA Inc.
San Jose, CA8 hours ago$342.7k
...Enablement team is the engine behind Cisco’s... .... Our mission is to build and scale CIRCUIT, Cisco... ...through intelligent, agentic experiences... ...agentic systems, multi-agent orchestration, LLM integration, search... ...memory frameworks, evaluation and reliability systems...Full timeTemporary workLocal areaFlexible hours$156.85k - $187.68k
...physical and digital worlds to enable breakthroughs at the Intelligent Edge. ADI combines analog, digital, AI, and software... ...LinkedIn and X.Employer: Analog Devices, Inc. Job Title: Senior Engineer, Design Evaluation EngineeringJob Requisition: R262582Job Location: San Jose...Permanent employmentFull timeWork at officeLocal areaRemote workWork from homeDay shift2 days per week
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to LLM Agent Evaluation Engineer — Build Intelligent Agents. Be the first to apply!
Related searches