LLM Agent Evaluation Engineer — Build Intelligent Agents
Bytedance
ByteDance Applied Machine Learning Ark team in San Jose develops and operates LLM service platforms that offer MaaS solutions to businesses and researchers, spanning model serving, tool calling, and analytics. This full-time role emphasizes designing evaluation systems for LLM agents, building benchmarks and pipelines, analyzing traces and feedback, and collaborating with research, platform, and product teams to bring methods into production. #J-18808-Ljbffr Bytedance
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the LLM Agent Evaluation Engineer — Build Intelligent Agents in San Jose, CA vacancy
- ...California's San Jose area seeks engineers and researchers for the... ...Machine Learning Ark team to build MaaS platforms and end-to-end LLM services, spanning training, inference... ...The role emphasizes designing evaluation systems for LLM-based agents, building benchmarks, and...Suggested
$175k - $287k
...exciting opportunities, build necessary skills,... ...is building the Evaluation Operating System (... ...), a foundational Agent Evaluation... ...acts as the central intelligence layer for AI quality... ...evaluator models (e.g., LLM-as-a-judge, reward... ...Agents.As a Staff Engineer, you will own the...SuggestedFor contractorsWork experience placementWork at officeFlexible hours- ...develops end-to-end MaaS platforms that combine system engineering with large language models to serve businesses globally... ...with a solid ML background, Python skills, and hands-on LLM experience to build evaluation pipelines, benchmarks, and production-grade components...Suggested
$152k - $241.5k
...world working for us. If you're passionate about building reliable systems software for cloud-scale GPU... ...our team. We are looking for a Senior Software Engineer to join our DGX Cloud / Fleet Intelligence team and build agent-side systems that collect GPU health, host telemetry...SuggestedFull timeLocal areaRemote work$184k - $287.5k
Artificial intelligence is shifting from passive help to autonomous... .... We seek a Senior Engineer to lead technical... ...deploying advanced AI agent frameworks and local runtimes... ...environments. You will build the technical roadmap... ...understanding of LLM inference pipelines (Ollama...SuggestedFull timeLocal areaShift work- ...platforms that span text and multimodal capabilities. You will help build end-to-end MaaS solutions, data pipelines, and evaluation systems across US and international markets. The internship blends system engineering with cutting-edge AI research, offering hands-on experience,...Internship
$162k - $316.8k
...team combines system engineering and machine learning... ...Large Language Model (LLM) service platforms that... ...China. We are building full-stack, end-to-end... ...model alignment, and intelligent agent systems. Beyond model... ...dynamic team. Design evaluation systems for LLM-based...Temporary workInternshipLocal area- ServiceNow in Santa Clara, CA is seeking a leader for the Build Agent evaluation framework. You will own eval strategy, roadmap, telemetry, and cross-team quality standards, guiding an 8‑engineer team to deliver scalable, data‑driven improvements. You will benchmark models...
$187.04k
...team. Our mission is to build the foundational AI agent platform and intelligent automation workflows... ...safety needs into scalable engineering solutions.... ...experience integrating LLM/ML capabilities into products... ...handling model latency, evaluation flows). Prior experience...Temporary workLocal areaShift work- Otter.ai is seeking a Senior AI Agent & LLM Engineer in Mountain View, CA. You will build and improve AI systems behind Otter's conversational knowledge engine... ...AI Chat, spanning user-facing agent experiences, evaluation systems, and the shared platform for scalable, reliable...
- A forward-thinking technology firm in San Jose is looking for an LLM Engineer to design and deploy large language models that drive innovative AI solutions. The role includes designing models, building inference pipelines, and collaborating with cross-functional teams....
$171.6k - $222.2k
...world-class scientists and engineers to pioneer the next... .... Join the Amazon Kiro LLM-Training team and help... ...models specialized in code intelligence- Invent and implement... ...teamThe AWS Developer Agents and Experiences (DAE) team... ...time to focus on a build. We also value in-person...Work at officeLocal areaWorldwideFlexible hours- Nutanix is seeking a software engineer based in Santa Clara, California, to develop AI-powered product features. The role requires building backend services and APIs, implementing LLM workflows, and collaborating with product managers and designers. The ideal candidate...
- ServiceNow is seeking a Software Engineering Manager for Build Agent to lead an 8-engineer team focused on evaluation infrastructure, model benchmarking, and telemetry. The role drives strategy, cross-team collaboration, and scalable processes to ensure robust evaluation...
$155k - $185k
...are looking for a Senior AI Agent & LLM Engineer who combines strong... ...AI quality. You will help build and improve the AI systems... ...-facing agent experiences, evaluation systems, and the shared platform... ...collaboration. Using artificial intelligence, Otter generates real-time...Permanent employment$212.8k - $387.6k
...projects, focusing on LLM/AI + Infrastructure technologies... ...To name a few, we are building a new cloud-native... ...Models (LLMs) And AI Agents, Traditional Cloud-native... ...vector indexing engine to support ultra-large-... ...also our research focus. Intelligence & Agent Architecture Explore...Temporary workLocal area$139k - $257.55k
...Collaboration, focuses on building foundational... ...create together intelligently and efficiently.... ...users and AI agents. Our mission is to... ...forward-thinking engineers who are excited to... ...Design and integrate LLM-powered and... ...experience.Develop evaluation and quality frameworks...Full timeTemporary workLocal areaWorldwide$42.75 per hour
...Payment team at TikTok builds the core payment... ...the Payment Data Intelligence team applies data... ...on building agent-first AI applications... ...paradigms, including LLM-based agents,... ...orchestration, prompt engineering, MCP-based tool integration... ...AI application evaluation, to solve...Hourly paySummer workInternshipLocal areaShift work- ...highly experienced AIML - Machine Learning Engineer to join the Information Intelligence team in Cupertino. You will conduct research, build and refine domain-specific LLMs for... ...focuses on large-scale model training, evaluation, and deployment, with opportunities to influence...
$166.5k - $291.4k
...DescriptionIt all started when engineer Fred Luddy wrote code... ...inspired Fred to build a company that could do... ...DescriptionThe teamBuild Agent is ServiceNow's AI... ...responsible for the Build Agent evaluation framework, model... ...evaluation systems for LLM-based products — scoring...Work at officeImmediate startRemote workFlexible hours$185k - $230k
Senior Software Engineer, AI Agent & LLM Mountain View, CA The Opportunity We... ...AI quality. You will help build and improve the AI systems... ...facing agent experiences, evaluation systems, and the shared platform... .... Using artificial intelligence, Otter generates real-time...Permanent employmentContract workFor contractorsFor subcontractorWork at office$224k - $356.5k
Artificial intelligence is moving from passive assistance to agents that can reason, use tools... .... NVIDIA is building the software foundation... ...looking for an Engineering Manager to build... ....Own CI/CD, evaluation, compatibility, and... ...Experience with TensorRT-LLM, Ollama, llama....Full timeLocal area$184k - $287.5k
In the NVIDIA Isaac team, we build tools that bring the power of the... ....We’re looking for a research engineer to join our team to develop... ...the industry's leading tool for evaluating robot foundation models in... ...foundation models.Integrating modern LLM and agentic workflows into...Full time$10,000 per month
...Description Job Description Engineer in Residence, Enterprise Agents Large companies run on... ...systems in front of an LLM stack without a months-... ...cycle. This role is to build secure AI agents that... ...We may use artificial intelligence (AI) tools to support parts...Full timeContract workVisa sponsorshipcareer
Mountain View, CA7 days ago- TikTok's Commercial AI-CRM and Transaction team focuses on advertiser growth algorithms, leveraging LLMs and Agent tech to build intelligent systems for business objectives, growth, and decision‑making. You will contribute to enterprise‑grade AI initiatives, exploring memory...Internship
- ...is offering a Machine Learning Engineer Intern position for the 2027... ...and large language models to build scalable AI systems. You will explore Enterprise AI Agent technologies, advanced dialogue... ...and AI call capabilities, and intelligent decision‑making with practical...Summer workInternship
$184k - $287.5k
...We are now looking for a Senior High-Performance LLM Training Engineer!NVIDIA is seeking experienced engineers specializing in performance analysis... ...deep learning platform stack, from drivers to DL frameworks.Build and support NVIDIA submissions to the MLPerf Training...Full timeWork experience placement$184k - $287.5k
...seeking an AI Compiler Engineer with deep expertise... ...role is focused on building AI-driven compiler intelligence for production... ...efficiency, and advancing LLM-enabled workflows... ...experimentation, evaluation, and deployment... ...LLM-based systems, agents, or developer tooling...Full timeRemote work$138k - $206k
...selves. Together, we’re building a better tomorrow for... ...(Artificial General Intelligence) Computing Lab is dedicated... ...hardware and software engineers to identify and... ...for modern and emerging LLM workloads.We are seeking... ...resources, and evaluate how emerging workloads...Work experience placementWork at officeFlexible hours$272k - $431.25k
...NVIDIA is seeking a Principal Engineer to drive the performance of large-scale AI training... ...will analyze and optimize frontier-scale LLM workloads running on thousands of GPUs, drive... ...across NVIDIA’s AI software stack.Build and refine performance models, workload characterizations...Full time
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to LLM Agent Evaluation Engineer — Build Intelligent Agents. Be the first to apply!
