ML Lead, AI Data Labeling

About NewtonX

NewtonX is a B2B insights company trusted by the worlds most innovative companies to make high-stakes decisions with confidence. We combine a verified network of business professionals with AI-powered research tools to deliver research intelligence faster, more precise, and more defensible than traditional methods. Our clients include Google, Microsoft, TikTok, DoorDash, Stripe, and Coinbase. Our research has been cited by Fortune, Forbes, TechCrunch, Adweek, and the Wall Street Journal. NewtonX has raised $47M from investors including Two Sigma Ventures, Third Prime, XFund, and Citi Ventures.
About the Role
AI buyers have changed. From mid-market SaaS companies fine-tuning open-source models to Fortune 500 enterprises building internal AI platforms to frontier AI labs running large-scale evaluations, the question is no longer “is AI useful” but “how do we evaluate whether our AI works?” Every one of these buyers needs structured, expert-grounded evaluation data and domain-specific benchmarks. Almost none of them can build it themselves.
That is the opportunity as ML Lead. Rolling up directly to the VP of Commercial, you are the technical counterpart to ML and product teams across our client base, spanning growth-stage AI companies, enterprise AI platforms, and frontier research labs. You sit in their working sessions, hold your ground on technical specifics (eval design, statistical significance, contamination concerns, inter-annotator reliability), translate what they actually need into concrete operational specs, and partner with our recruiting and ops lead to build the expert pipelines that produce defensible data.
You also build. Beyond bespoke client work, you own the design and development of NewtonX domain benchmarks across high-value verticals (finance, legal, healthcare, and others as we expand). These become both syndicated products and methodological proof points that move us up the client sophistication curve.
And you sell, lightly but meaningfully. You are on client calls. You hear gaps. You spot opportunities other vendors miss. You bring those back, shape them into pitches, and partner with Commercial to expand accounts.
In this role youll focus on:

Client Technical Partnership
Serve as the primary technical point of contact for ML, applied science, and product teams at our AI-focused clients across the maturity spectrum, from emerging AI companies to enterprise platforms to frontier labs.
Hold your own in technical conversations: eval design, dataset construction, contamination risk, statistical power, inter-annotator agreement, RLHF data quality, agentic evaluation, red-teaming methodology.
Translate ambiguous technical requirements into concrete operational specs: target expert profiles, screener trees, task design, annotation rubrics, quality control protocols, statistical sampling plans.
Calibrate depth to the audience. A Series B AI startup and a frontier lab need different conversations. You can run both.
Domain Benchmark Development
Design and build domain benchmarks for NewtonX-owned domains in high-value verticals. Initial targets: finance (markets, accounting, regulatory), legal (contracts, case reasoning, jurisdictional), healthcare (clinical reasoning, diagnostic, regulatory). Additional verticals as the business expands.
Architect benchmark structure: task taxonomy, difficulty distribution, expert involvement model, evaluation rubrics, scoring protocols, baseline scoring against frontier models.
Recruit and calibrate the domain experts who write, validate, and grade benchmark tasks. Work with our recruiting and ops lead to operationalize at scale.
Publish methodology papers, technical reports, and leaderboards that make NewtonX benchmarks the reference standard in their verticals.
Operationalization with NewtonX Recruiting and Ops
Work directly with our full-time recruiting and operations lead to convert client and benchmark requirements into operational specs: expert profiles, screeners, task interfaces, annotation workflows, QC sampling rates, and fielding timelines.
Calibrate the recruiting team on what “good” looks like for each engagement. Run alignment sessions when standards shift.
Own the technical feedback loop: when an expert clears screening, but their output is unusable, diagnose whether it is a screener problem, a task-design problem, or a calibration problem, and fix it upstream.
Define quality control metrics: inter-annotator agreement targets, gold-standard task injection rates, and statistical power thresholds. Hold the team accountable to them.
Commercial Partnership and Account Expansion
Sit in client calls alongside Commercial leads. Surface technical gaps and unsolved problems that the client has not yet asked us to address.
Translate gaps into concrete proposal narratives: scope, methodology, deliverables, defensibility. Hand off to Commercial for pricing and close.
Contribute to NewtonX positioning with AI buyers: case studies, technical blog posts, conference presence at applied AI and industry events.
Help shape what additional ML and research roles we hire as the AI account book and benchmark program grow.
Who you are:

Required
5 to 8 years of applied ML experience with substantive evaluation, benchmark, or human data work. Examples of strong backgrounds: applied scientist or ML engineer who owned an eval or human data workstream; ML lead at an AI-forward enterprise who built domain-specific evaluation systems; research engineer at an AI consultancy or evaluation firm; quantitative researcher who pivoted into LLM evaluation.
Working fluency with modern LLM evaluation: benchmark design, contamination handling, statistical significance, eval harness construction, agentic and tool-use evaluation, RLHF and preference data quality, red-team probe design. You do not need to have built every one of these, but you should be conversant across them.
Strong programming foundation. You can read and reason about an eval harness, write Python comfortably, work with model APIs, and prototype scoring pipelines. You do not need to be a production engineer, but you should not be hands-off either.
Statistical fluency. You know when an effect is real and when it is noise. You can defend a sample size choice or a significance threshold.
Demonstrated client-facing presence. You have presented technical work to skeptical audiences, defended design choices in real time, and adjusted scope without losing rigor. Range matters here: you can talk to a Series B CTO and a Fortune 100 AI lead in the same week.
Light commercial instinct. You hear a client describe a problem, and your first reaction is, “We could solve that. Here is how.” You are comfortable shaping that into a pitch. You do not need to close, but you need to spot.
Strong written communication. You can write a methodology section, a benchmark technical report, or a client proposal that holds up to expert review.
Strongly Preferred
Direct experience designing or contributing to an LLM benchmark or evaluation system (academic, open-source, or proprietary).
Domain depth in one or more of: finance, legal, healthcare, scientific reasoning, and software engineering. Bonus for two or more.
Exposure to expert-driven data work: RLHF pipelines, preference data collection, expert annotation programs, red-team operations, and evaluation contractor management.
Graduate degree in computer science, machine learning, statistics, or a related quantitative field. A strong applied track record can serve as a substitute.
Publications or open-source contributions in evaluation, benchmarking, or applied ML methodology.
How we will evaluate

Technical screen: deep dive on a benchmark, eval system, or data pipeline you have built or contributed to. We will probe design choices, statistical reasoning, and what you would do differently.
Take-home exercise: We give you a real (anonymized) client problem and ask you to design an evaluation or benchmark, including task design, expert profile, sampling plan, scoring methodology, and quality-control protocol.
Live working session: walk through your take-home as if you were defending it to a client ML lead. We will push back. We want to see how you handle it.
Domain benchmark thought exercise: pick a vertical (finance, legal, or healthcare) and sketch what a defensible domain benchmark in that area would look like.
Cross-functional interviews with Commercial, our recruiting and ops lead, and senior leadership to assess collaboration, communication, and commercial instinct.
If the profile above describes you and your passions, wed love to hear from you!
What we offer

Massive Impact:

Opportunity to have an astounding impact, build a brand new business unit from the ground up, and have direct C-level influence at an extremely fast-growing late-stage startup.
Fast-track career growth:

This foundational role will enable you to progress quickly within NewtonX towards commercial and operational leadership.
Comprehensive Benefits:

Excellent medical, dental, and vision insurance.
Retirement:

401k match with immediate vesting.
Perks:

Health savings/flexible savings account, and pre-tax commuter benefits.
Work-Life Balance:

Paid time off: vacation, holidays, sick, and parental leave.
Great Culture:

A diverse, collaborative, and positive culture where we invest in and celebrate each others success (happy hours, team projects, and retreats).
Visa sponsorship is not available for this role.
NewtonX is proud to be an equal opportunity workplace. We do not discriminate based upon race, religion, color, national origin, sex, sexual orientation, gender identity/expression, age, status as a protected veteran, status as an individual with a disability, or any other applicable legally protected characteristics.

#J-18808-Ljbffr