Founding AI/ML Engineer, Agents

Job Description

Founding Applied AI Engineer, Agents & Evaluation

\n

\n

Compensation: $190,000–$250,000 base + equity

\n

\n

Location: New York City — fully onsite, 5 days per week

\n

\n

Employment Type: Full-time

\n

\n

Join a fast-growing seed-stage AI startup building intelligent systems for complex, high-stakes insurance investigations.

\n

\n

This team is building a claims intelligence platform that helps investigators uncover fraud across property and casualty insurance. By connecting information across claims, businesses, contractors, individuals, phone numbers, documents, and other data sources, the platform turns fragmented information into evidence investigators can act on.

\n

\n

The company has already built substantial backend, document-processing, data-extraction, and traditional machine learning infrastructure. With seven-figure customer agreements, the business is now investing heavily in the next layer of its platform: production-grade AI agents capable of carrying out sophisticated investigative workflows.

\n

\n

What You’ll Do

\n

\n
• Design, build, and iterate on production AI agents, including their tools, context, orchestration, prompts, and guardrails.
\n
• Own the evaluation framework for agent performance, from defining what “good” looks like through to measuring it systematically.
\n
• Partner directly with fraud investigators to create high-quality golden datasets based on real investigative workflows.
\n
• Build LLM-as-judge, deterministic, and rule-based graders to evaluate agent outputs.
\n
• Develop regression suites that validate prompt, model, and orchestration changes before they reach production.
\n
• Create quality dashboards that make agent performance and failure modes understandable across engineering, product, and operations teams.
\n
• Instrument production agents for traces, tool usage, cost, latency, and other critical performance metrics.
\n
• Turn real-world production failures into new evaluation cases and continuously improve the system.
\n
• Evaluate new foundation models and providers end-to-end, then own the technical rollout when the team decides to switch.
\n
• Partner closely with backend engineers on the data and service infrastructure agents depend on.
\n
• Work with Operations and investigators to understand how AI behaves in real, sometimes adversarial, investigative environments.
\n
• Build with a high bar for precision: incorrect findings can have genuine consequences for the people involved in a claim.
\n

\n

\n

What You’ll Bring

\n

\n
• Strong Python engineering skills and a disciplined approach to testing and software quality.
\n
• Hands-on experience shipping LLM-powered agents into production environments.
\n
• A clear understanding of why production agents succeed or fail beyond simply getting a prototype working.
\n
• Experience building evaluation datasets, graders, regression suites, or evaluation harnesses for LLM applications.
\n
• Evidence that evaluation results have directly influenced product, model, or engineering decisions you’ve made.
\n
• Strong knowledge of tool-using agents, agent orchestration, and emerging agent protocols and frameworks.
\n
• Experience moving comfortably between rapid experimentation and production hardening.
\n
• The judgment to know when a lightweight prototype is sufficient and when reliability, observability, and testing need to take priority.
\n
• A track record of personally owning meaningful production systems rather than stopping at demos or proofs of concept.
\n
• Strong communication skills and an interest in working directly with domain experts to turn ambiguous real-world workflows into measurable AI systems.
\n

\n

\n

Why Join?

\n

This is an opportunity to own the entire applied AI feedback loop: defining what a successful investigation looks like, building agents capable of performing that work, and creating the evaluation systems that demonstrate whether those agents are actually reliable.

\n

\n

You’ll be tackling a problem where quality goes far beyond benchmark performance. The environment is complex, investigative, and occasionally adversarial, making evaluation, observability, and engineering judgment central to the product rather than an afterthought.

\n

\n

The role offers substantial individual ownership and the chance to help define how a growing AI company builds, evaluates, and operates production agent systems from the ground up.

\n

\n

About People In AI

\n

People In AI is a specialist recruitment partner dedicated to connecting exceptional talent with some of the most ambitious companies building the future of artificial intelligence.

\n

\n

We work across Applied AI, Machine Learning, Generative AI, Data, Research, and AI Infrastructure, partnering with teams ranging from high-growth startups to established technology companies.

\n

\n

If you’re interested in building production AI systems where reliability, evaluation, and real-world impact genuinely matter, we’d love to hear from you.