Full-Stack Engineer, Agent Evaluation & Reliability

About the Role

Join a small, technical team building AI agent evaluation and reliability infrastructure for enterprise customers. This backend-leaning full-stack role focuses on making agent performance measurable and improving the systems that support reliable AI products.

What You'll Do

• Build backend systems and data models for evaluation datasets, experiment runs, agent traces, knowledge versions, and quality criteria.

• Develop and operate enterprise integrations, including authentication, data syncing, retries, monitoring, and production support.

• Ship features across the stack, from technical design and implementation through deployment, monitoring, and iteration.

• Work with customers to diagnose system failures and turn their needs into reusable product capabilities.

• Build data pipelines and interfaces for collecting, analyzing, and visualizing evaluation results.

What We're Looking For

• At least 3 years of experience building and shipping production systems as a full-stack engineer, ideally including work at a fast-paced startup.

• Hands-on machine learning model development experience, along with knowledge of evaluating and testing AI agents or other nondeterministic systems.

• Strong backend skills with TypeScript and Node.js or Bun, and experience building production frontend features.

• Strong PostgreSQL skills, including query optimization, and practical experience with Redis, job queues, background workers, or workflow systems.

• Experience designing APIs and backend services, building evaluation data pipelines, and developing dashboards or visualization interfaces.

• Familiarity with observability, monitoring, logging, and production troubleshooting.

Compensation & Benefits

Base salary: $120,000 to $180,000 annually. Visa sponsorship is available.

Location

On-site in San Francisco, California, United States.