Senior Data Scientist (Model Diagnostics)

You'll own the investigative science behind the core prediction models at a mobile advertising technology company: finding where models underperform, figuring out why, and turning that into the roadmap the ML engineering team builds against. This isn't a model-building or infrastructure role. You'll diagnose, experiment, and explain, and your work will steer a multi-billion-dollar optimization problem.

What you'll do

• Dig into where install-prediction models lose accuracy, slicing by segment, data slice, and campaign type to isolate where and why performance drops
• Run calibration checks, feature-importance analysis, and slice-level error analysis to turn a vague performance gap into a precise, testable explanation
• Form hypotheses about model behavior and test them empirically (for example, rerunning a pipeline on a reduced sample to test a data-volume theory)
• Translate findings into prioritized recommendations for ML engineering, defining the experiment and the expected mechanism, not just the metric
• Own production investigations end to end: when a campaign's performance drops, find the root cause (data issues, model shift, prediction drift, privacy-driven signal loss) and report back
• Surface longer-term opportunities across the model portfolio, quantifying the potential lift and trade-offs
• Keep technical leadership updated with status and new hypotheses without being asked

What you bring

• 5–7+ years in data science or applied machine learning, with hands-on model diagnostics work
• Strong applied statistics and probability, including experiment design
• Model calibration and error analysis on production models
• Python and SQL
• Master's in data science, statistics, applied math, computer science, or a related quantitative field
• Comfort explaining why a change moved a metric in terms of weights, loss, and predictions
• Self-directed: you take a loosely defined problem and drive it to an answer

Nice to have

• PhD
• Ad tech or large-scale real-time prediction systems
• Class imbalance and feature importance analysis

The details

• Hybrid, New York City (onsite 2–3 days a week)
• Full-time, permanent
• Senior individual contributor / technical lead, no direct reports
• U.S. Eastern hours, with daily overlap with a Europe-based team
• Immediate start