LLM Agent Evaluation & Evolution Architect
ByteDance
ByteDance's Applied Machine Learning Ark team develops end-to-end MaaS platforms that combine system engineering with large language models to serve businesses globally. The US team designs and operates MaaS solutions across the US and international markets beyond mainland China. We seek PhD-level researchers and engineers with a solid ML background, Python skills, and hands-on LLM experience to build evaluation pipelines, benchmarks, and production-grade components for scalable AI systems. #J-18808-Ljbffr ByteDance
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the LLM Agent Evaluation & Evolution Architect in San Jose, CA vacancy
- ...Machine Learning Ark team in San Jose develops and operates LLM service platforms that offer MaaS solutions to businesses... ..., and analytics. This full-time role emphasizes designing evaluation systems for LLM agents, building benchmarks and pipelines, analyzing traces and...SuggestedFull time
$162k - $316.8k
...develop and operate Large Language Model (LLM) service platforms that offer businesses... ...engineering, model alignment, and intelligent agent systems. Beyond model serving, we operate... ...to join our dynamic team. Design evaluation systems for LLM-based agents, covering task...SuggestedTemporary workInternshipLocal area- ...Machine Learning Ark team to build MaaS platforms and end-to-end LLM services, spanning training, inference, and multi-model workflows. The role emphasizes designing evaluation systems for LLM-based agents, building benchmarks, and turning failure patterns into concrete...Suggested
- NVIDIA is seeking a Senior Software Engineer for Agent Architecture and Evaluation to build agentic AI systems and analyze inference workloads in realistic settings. You will work with teams advancing evaluation across software and hardware to improve accuracy and efficiency...Suggested
- ...Job Overview We are looking for an AI Agent Architect to design and build enterprise internal... ...delivery plans, then improve them through evaluation and production feedback. What We... ...systems. Hands-on understanding of LLM and agent systems, including tool use,...Suggested
Maxinsights
Santa Clara, CA3 days ago $212.8k - $387.6k
...teams on advanced R&D projects, focusing on LLM/AI + Infrastructure technologies, which... ...Of Large Language Models (LLMs) And AI Agents, Traditional Cloud-native Infrastructure... ...enabling elastic scalability, and driving the evolution of AI infrastructure technologies. Topic...Temporary workLocal area- Maxinsights is seeking an AI Agent Architect to design and build enterprise internal agents and AI-native workflows. You will own the architecture... ...into scalable designs and delivery plans, then refine them through evaluation and production feedback. #J-18808-Ljbffr Maxinsights
$184k - $287.5k
We are now looking for a Senior Deep Learning Architect, LLM Inference!NVIDIA is at the forefront of the generative AI revolution. The Inference... ...to ensure best-in-class performance.Use the latest coding agents and inference technology to improve team efficiency.What we...Full time$152k - $241.5k
...looking for a Senior Performance Architect for Nemotron! At NVIDIA, we... ...high-fidelity models to evaluate how architectural choices translate... ...the center of Generative AI evolution, partnering across research,... ...frameworks like PyTorch, TRT-LLM, VLLM, SGLangA Growth mindset...Full time$171.6k - $222.2k
...generation of AI-driven development tools. Join the Amazon Kiro LLM-Training team and help create groundbreaking generative AI technologies... ...into production systemsAbout the teamThe AWS Developer Agents and Experiences (DAE) team is reimagining the builder experience...Work at officeLocal areaWorldwideFlexible hours- ...service platforms that span text and multimodal capabilities. You will help build end-to-end MaaS solutions, data pipelines, and evaluation systems across US and international markets. The internship blends system engineering with cutting-edge AI research, offering hands...Internship
$152k - $241.5k
...Demand for inference is growing rapidly, and coding and research agents are becoming major drivers of token usage. New agentic patterns... ...under realistic conditionsCollaborating with teams that lead evaluation systems for software and hardware to improve eval qualityWhat we...Full time$155k - $185k
The OpportunityWe are looking for a Senior AI Agent & LLM Engineer who combines strong software engineering capabilities with a deep focus... ...engine and AI Chat, spanning user-facing agent experiences, evaluation systems, and the shared platform required to operate them...Permanent employment$152k - $241.5k
We are now looking for a Senior Software Engineer for Agent Simulation & Evaluation! Today, NVIDIA is tapping into the unlimited potential of AI to define the next era of computing. An era in which our GPUs act as the brains of computers, robots, and self-driving cars that...Full time- ...research emerging AI/ML technologies and collaborate with data engineers and software teams to deliver end-to-end solutions. You will evaluate performance, fine-tune models for scalability, and mentor junior engineers while communicating complex results to stakeholders....
- Lorven Technologies Inc. in San Jose, CA seeks a senior AI/ML engineer with 6-12 years of experience in AI/ML/deep learning model development, plus 5 years on cloud AI platforms (AWS SageMaker, Azure ML, GCP AI). The role requires 2-3 years with LLMs, NLP, or speech/voice...
- ...initiatives, including low-latency scoring, automated evaluation, and RAG integrations. You will guide... ...product and data science, and help scale agent-based commerce platforms globally. The role emphasizes building reliable LLM tooling, Ground Truth datasets, and dashboards...
$85 per hour
Responsibilities About the team The Seed LLM Post Training team is responsible for researching cutting‑edge posttrain technologies and... ...optimizing and improving key areas including reasoning, coding, agent, and omni model. Responsibilities Explore large‑scale models and...Hourly payInternship$184k - $287.5k
...openings for a Deep Learning Communication Architect. We scale the DNN models and training/... ...communication technologies: Research and evaluate new communication technologies and techniques... ...in evaluating, analyzing, and optimizing LLM training and inference performance of...Full timeWork experience placement$208k - $327.75k
...architectures.We are looking for a Senior AI Architect to help define the next generation of AI... ...and performance analysis.Prototype and evaluate emerging model paradigms on NVIDIA DRIVE... ...Jetson, CUDA, TensorRT, Triton, or TensorRT-LLM.Experience influencing silicon...Full timeWorldwide$206.4k - $379.1k
...We're looking for a Principal Architect to build and implement the AI... ...analytics, and continuous evaluation frameworks.This role blends applied... ..., data pipelines, LLM orchestration layers, in-house... ...task decomposition, and multi-agent coordination.Skill to merge engineering...Full timeTemporary workLocal areaWorldwideFlexible hours$100k - $160k
...RoleWe are seeking a Senior Mainframe DevOps Architect to lead solution design and own end-to-... ...through critical technical evolutions, this position offers the scope and autonomy... ...across the customer’s IT leadership silos to evaluate complex environments (z/OS, security frameworks...Full timeLocal areaShift work- ServiceNow in Santa Clara, CA is seeking a leader for the Build Agent evaluation framework. You will own eval strategy, roadmap, telemetry, and cross-team quality standards, guiding an 8‑engineer team to deliver scalable, data‑driven improvements. You will benchmark models...
$185k - $230k
Senior Software Engineer, AI Agent & LLM Mountain View, CA The Opportunity We are looking for a Senior AI Agent & LLM Engineer who combines... ...engine and AI Chat, spanning user-facing agent experiences, evaluation systems, and the shared platform required to operate them...Permanent employmentContract workFor contractorsFor subcontractorWork at office- Build agentic systems to automate simulation, evaluation, and performance analysis for AI workloads across GPU and LPU platforms. Collaborate... ...experience, including 2 years specifically building AI agents. Candidates must be proficient in Python and have experience with...
- ...ML Accelerator product. As an AI/ML ASIC Architect you will help drive new architecture initiatives... ...RTL/DV/Simulation/Emulation/FW teams to evaluate these changes and assess the performance,... ...during the execution phase. LLM Workload analysis and characterization of...Temporary workRemote workFlexible hoursShift workNight shift
SanDisk
Milpitas, CA5 days ago $219k - $351k
...engineer, AI Serving Framework Architect (Software)The Architecture... ...operations in RAG’s vector DB and AI Agent’s knowledge-graph by... ...optimize a Large Language Model (LLM) Inference Software Stack on a... ...to ensure every candidate is evaluated fairly and holistically.Recruiting...Work at officeFlexible hours$175k - $287k
...LinkedIn’s Core AI is building the Evaluation Operating System (EOS), a foundational Agent Evaluation platform that defines... ..., evaluator models (e.g., LLM-as-a-judge, reward models), and real... ...with trace debuggability features.Architect scalable data pipelines and platforms...For contractorsWork experience placementWork at officeFlexible hours$224k - $356.5k
...intelligence is moving from passive assistance to agents that can reason, use tools, and complete... ...local or hybrid model routing.Own CI/CD, evaluation, compatibility, and regression systems... ...-use patterns.Experience with TensorRT-LLM, Ollama, llama.cpp, vLLM, PyTorch, ONNX...Full timeLocal area$160k - $200k
...to design and ship the production multi-agent systems at the core of LeanData’s new platform... ...at enterprise scale — including the evaluations that prove they work and the path that safely... ...systems, including 2+ years shipping LLM-powered or agentic systemsYou have shipped...Work at office
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to LLM Agent Evaluation & Evolution Architect. Be the first to apply!
