Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Principal Data Scientist - Experimentation & Causal Inference

ByLabs

Job Description

About the Role

We are looking for a Data Scientist - Experimentation & Causal Inference to serve as the statistical brain behind our rapidly growing experimentation program. You will own the methodology layer that ensures every experiment we run is trustworthy, sensitive, and correctly interpreted.

You will NOT be building infrastructure or writing backend services — our platform engineering team handles that. Your job is to design, validate, and continuously improve the statistical frameworks that sit on top of the platform. You will be the person the team turns to when they ask "Can we trust this result?" — and you will build the automated systems that answer that question before anyone needs to ask.

Team context: You will join a team of data scientists and platform engineers within the Big Data group, reporting to the Head of Big Data. The experimentation platform engineering team implements your specifications into production systems.

Responsibility

Guardrail Metrics & SRM Detection

  • Design and maintain automated anomaly detection for live experiments — including Sample Ratio Mismatch (SRM) checks and traffic split validation.
  • Define alerting thresholds and circuit-breaking criteria so compromised experiments are flagged or stopped before polluting decisions.
  • Define and validate guardrail metrics (sensitivity, directionality, interpretability) that protect the business during every experiment.

Variance Reduction & Sensitivity

  • Implement and iterate on CUPED and related pre-experiment covariate adjustment methods to reduce metric variance.
  • Develop techniques to remove noise from user historical behavior, enabling faster detection of true treatment effects — especially in limited-traffic or high-priority scenarios.
  • Example of impact we are targeting: Shorten average experiment duration from 14 days to 9 days, unlocking 40%+ more experiments per quarter.

Continuous A/A Experiment Monitoring

  • Design and run continuous A/A experiments as an always-on health check for the data pipeline (from client-side event reporting through message queues to the real-time data warehouse).
  • Monitor metric baseline volatility and pipeline stability, ensuring the instrumentation layer remains trustworthy over time. You define what "healthy" looks like; the platform team implements the monitoring infrastructure.

Other responsibilities

  • Partner with product, engineering, and growth teams on experiment design: sample size calculations, metric selection, duration estimation, and result interpretation. Advise on causal inference methods (DID, synthetic control, RDD) when randomization is not feasible.
  • Translate your methods into specifications, validation scripts, and decision frameworks that the team can operationalize — enabling experiment owners to self-serve while maintaining rigor.

Requirements

  • Education: MS or PhD in Statistics, Biostatistics, Economics (Econometrics), Computer Science, or a related quantitative field.
  • Experience: 5+ years of industry experience designing and analyzing online controlled experiments at a tech company with meaningful user scale (not exclusively survey experiments or clinical trials).
  • Core Statistical Knowledge: Solid foundations in hypothesis testing, power analysis, multiple testing correction, and sequential testing. Hands-on experience with at least two of: variance reduction methods (CUPED or similar), sample ratio mismatch detection, or continuous data quality monitoring for experiments.
  • Programming: Proficient in Python (scipy, statsmodels, or equivalent) for statistical analysis and simulation. Comfortable writing complex SQL (window functions, CTEs) for data extraction and validation.
  • Communication & Collaboration: Ability to explain complex statistical concepts to non-technical stakeholders, translate business questions into rigorous experimental designs, and define clear specifications so engineers can implement your methods in production.
  • AI-Native Workflow: Strong AI Sense — extensive hands-on experience with AI coding tools (Claude Code, OpenClaw, or similar). Demonstrated ability to leverage AI assistants to automate repetitive analytical tasks, accelerate code development, and build self-serve tooling faster. You treat AI tools as a daily productivity multiplier, not a novelty.

Nice-to-Haves

  • Familiarity with Bayesian experimentation frameworks or multi-armed bandit approaches.
  • Understanding of real-time streaming data pipelines (Kafka, Flink) — not as an engineer, but how data flow affects statistical validity.
  • Publications or conference presentations on experimentation methodology or causal inference (KDD, WSDM, CIKM, NeurIPS CausalML workshop, or equivalent).
  • Experience in fintech, crypto, or high-frequency trading domains where user behavior has high variance and fast-moving baselines.

Vacancy posted 18 days ago
Similar jobs that could be interesting for youBased on the Principal Data Scientist - Experimentation & Causal Inference in Seattle, WA vacancy
  •  ...About the Role We are looking for a Data Scientist - Experimentation & Causal Inference to serve as the statistical brain behind our rapidly growing experimentation program. You will own the methodology layer that ensures every experiment we run is trustworthy, sensitive... 
    Principal
    Full time

    ByLabs

    Seattle, WA
    1 day ago
  • $110k - $220k

     ...Home OfficeRole summaryAs a Principal Data Scientist at Walmart, you will...  ...data science roadmap for the experimentation platform that powers trusted...  ...on experimental design, causal interpretation, metric tradeoffs...  ...experimentation, causal inference, and statistical decision-... 
    Principal
    Full time
    Temporary work
    Part time

    Walmart

    Bellevue, WA
    4 days ago
  •  ...Job Title: Data Scientist IV (Product Analytics & Experimentation) Location: Bellevue, WA Duration: 12 Months (Potential...  ...testing, and statistical inference. ~ Experience defining and monitoring...  ...distributed datasets. Familiarity with causal inference, experimentation... 
    Suggested

    Intelliswift - An LTTS Company

    Bellevue, WA
    9 days ago
  • $132k - $264k

     ...develop scalable, robust technical solutions. The data scientist role will be part of the data science team and...  ..., unsupervised models, optimization, graph ML, causal inference, causal ML, statistical learning, experimentation, and Gen-AI.In Gen-AI, it is desirable to have... 
    Principal
    Full time
    Temporary work
    Part time
    Flexible hours

    Walmart

    Bellevue, WA
    4 days ago
  • $203.5k - $299.3k

     ...the next generation of causal decisioning systems...  ...models, surrogate metrics, experimentation platforms,...  ...experimentation, observational data, and ML decisioning so...  ...experience with causal inference, econometrics,...  ...engineers, economists, data scientists, product managers, and... 
    Suggested
    Hourly pay
    Work at office
    Local area
    Remote work
    Flexible hours

    Doordash

    Seattle, WA
    4 days ago
  • $189.4k - $256.2k

     ...want to push the boundaries of data science (e.g. statistical modeling, causal inference, econometrics, product...  ...looking for a customer-focused Principal Data Scientist to lead and define the science...  ..., causal inference, and experimentation systems- Experience leading... 
    Principal
    Flexible hours

    Amazon

    Seattle, WA
    4 days ago
  • $282.1k - $414.8k

     ...next generation of causal decisioning systems...  ...question.We are hiring a Principal Machine Learning...  ...engineers and scientists through technical direction...  ..., observational data, policy logs,...  ...+ years) in causal inference, econometrics, experimentation, or causal machine... 
    Principal
    Hourly pay
    Work at office
    Local area
    Immediate start
    Remote work
    Flexible hours

    Doordash

    Seattle, WA
    11 hours ago
  • $172.5k - $313.7k

     ...are the future of Salesforce.We are hiring Staff & Principal Product Data Scientists to help build a world-class data ecosystem that...  ...highly desirable.Strong foundation in statistics, experimentation, causal inference, ML, and analytical problem-solving.Experience working... 
    Principal
    Full time

    Salesforce

    Seattle, WA
    4 days ago
  • $110k - $220k

     ...PermanentCompany: WalmartWe are seeking a Principal Data Scientist to provide technical leadership and...  ...techniques, feature engineering, and experimentation design - raising the technical bar...  ...statistical analysis, experiment design, and causal inferenceExperience deploying... 
    Principal
    Full time
    Temporary work
    Part time
    Home office

    Walmart

    Bellevue, WA
    2 days ago
  • $217.8k - $294.7k

     ...carries consequences for customers and advertisers.As Principal Data Scientist, you will own the analytical rigor behind Amazon's store...  ...high-visibility problems. You are fluent in statistical inference, experimental design, and sampling methodology, and you write... 
    Principal
    Local area
    Flexible hours

    Amazon

    Seattle, WA
    4 days ago
  • $193.4k - $420.2k

     ...accuracy grounded in the richest enterprise data and process context in the world. As a Data and Applied Scientist at SAP, you'll build the context engine grounded...  ...analysis and forecasting, anomaly detection, causal inference, operations research, mathematical... 
    Principal
    Permanent employment
    Full time
    Worldwide
    Flexible hours

    SAP

    Bellevue, WA
    4 days ago
  • $293k - $325k

     ...OpenAI builds and operates the experimentation platform that powers product development...  ...of experimentation methodology, data infrastructure, causal inference, and product analytics. We are...  ...We are hiring a Staff-level Data Scientist to help lead the evolution of OpenAI... 
    Remote job
    Full time

    openai

    Seattle, WA
    8 days ago
  • $293k - $325k

     ...The Statsig team at [OpenAI]( builds and operates the experimentation platform that powers product development, measurement,...  ...operates at the intersection of experimentation methodology, data infrastructure, causal inference, and product analytics. We are looking for experienced... 
    Full time

    OpenAI

    Seattle, WA
    8 days ago
  • $168k - $220.5k

     ...environments and move with an experimental mindset — who rapidly test...  ...work gets done.The Product Data Science team is looking for a Full-stack Senior Data Scientist to come aboard and be part of...  ...and extensible analytics/ML/causal inference frameworks to uncover feature... 
    Flexible hours

    Snowflake

    Bellevue, WA
    4 days ago
  •  ...-art AI agent technologies across the data science workflow. Design and productionize...  ...opportunity identification through experimentation, iteration, and impact measurement.3....  ...and simulation.4. Use experimentation, causal inference, and rigorous measurement frameworks... 
    Worldwide

    TikTok

    Seattle, WA
    11 hours ago
  • $164.43k - $287.75k

     ...use AI in our recruiting process here.We are looking for Data Scientists to join our Monetization Engineering organization. You...  ...velocity and quality using data scientist tool kit: experimentation, causal inference techniques, etc. Design measurement strategy, advise on... 
    Local area
    Relocation
    Relocation package

    Pinterest

    Seattle, WA
    4 days ago
  •  ...Data Scientist – Global Communications Measurement & Insights | Freelance/B2B | 12-Month Contract | 3 days on-site in Seattle...  ...channels. Using advanced statistical analysis, experimentation and causal inference to uncover what drives business outcomes. Designing... 
    Contract work
    Freelance

    YunoJuno

    Seattle, WA
    2 days ago
  •  ...is looking for an experienced Senior Data Scientist to lead brand analytics at DoorDash....  ...techniques and hypothesis testing or causal inference to validate your findingsProvide insights...  ...analysis (e.g. hypothesis testing, experimentation, regressions) with statistical... 
    Hourly pay
    Work at office
    Local area
    Remote work
    Flexible hours

    Doordash

    Seattle, WA
    1 day ago
  • $166.04k - $171.04k

     ...offer reliable connectivity for all. Sr Data Scientists is located in Bellevue, WA will...  ...Marketing Platform and Adobe Analytics, and causal inference methodologies: Bayesian structural...  ...validation, hypothesis testing, and experimentation frameworks, utilizing Python and R-... 
    Full time
    Temporary work
    Part time
    Work experience placement
    Local area
    Remote work
    Flexible hours
    3 days per week

    T-Mobile

    Bellevue, WA
    11 hours ago
  • $165k - $185k

     ...com:We're seeking a Senior Data Scientist to serve as an embedded Data...  ..., and deploying real-time inference systems that make decisions...  ...A/B testing, and iteration.Experimentation & Performance MeasurementDesign...  ...experiments to measure the causal impact of model changes on... 
    Temporary work
    Home office
    Flexible hours
    Shift work

    Impact Tech

    Seattle, WA
    2 days ago
  • $138.8k - $194.3k

     ...GETTING STARTED.Job Title: Senior Data Scientist, Insurance PricingLocation:...  ..., lightgbm)Experience with experimental design, including building,...  ...environments, as well as causal analysisExperience with cloud...  ...frameworks and causal inference methodsNice to have: experience... 
    Full time
    Temporary work
    Work experience placement
    Remote work
    Flexible hours

    Porch Group

    Seattle, WA
    4 days ago
  • $189.4k - $256.2k

     ...throughout the content discovery journey. We are looking for a customer-focused, solutions-oriented Principal Data Scientist to develop next-gen measurement and experimentation systems within Prime Video Personalization and Discovery. You'll be part of an embedded science... 
    Principal
    Temporary work
    Flexible hours

    Amazon

    Seattle, WA
    4 days ago
  •  ...suite of incrementally measurement products. We are seeking an experienced data scientist to focus on building digital ad measurement products by applying experimental design and causal inference approaches to help advertisers measure their true business value on the... 

    TikTok

    Seattle, WA
    1 day ago
  •  ...ensure quality and trust at scale. Our GNE Data Science & Analytics team drives a...  ...impacts;- Conduct A/B testing and complex experimentation (bilateral, networks effect, etc.);-...  ...3 years of Data Science experience in Causal Inference, Experimentation, Product Analytics,... 

    TikTok

    Seattle, WA
    1 day ago
  • $136k - $184k

     ...shape? We're looking for a Data Scientist who thrives on ambiguity and...  ...measurement, modeling, and experimentation across how customers...  ...forecasting demand, and applying causal methods to understand what...  ...Experience applying causal inference methods (e.g., experimentation... 
    Immediate start
    Flexible hours
    Day shift

    Amazon

    Seattle, WA
    11 hours ago
  •  ...people and we’re just getting started.Data ScientistLocation We're looking for...  ...'re looking for an experienced Data Scientist to tackle important, ambiguous...  ...You'll combine statistical reasoning, experimentation, causal inference, and business judgment to shape strategy... 
    Work at office

    Opendoor

    Seattle, WA
    1 day ago
  •  ...About the role As part of our growing Data Science and Analytics team, you will...  ...Develop hypotheses and apply rigorous causal inference methods — including controlled experiments...  ...data visualization tools Expertise in experimental design, causal inference, statistical... 
    Full time
    Work at office
    Visa sponsorship
    Flexible hours

    Anthropic

    Seattle, WA
    16 days ago
  • $304k

     ...dynamic and fast-moving environments and move with an experimental mindset — who rapidly test emerging capabilities to...  ...the future of how work gets done.We are hiring a Principal Engineer II to architect the core data processing engine of the Snowflake Data & AI Cloud.... 
    Principal

    Snowflake

    Bellevue, WA
    4 days ago
  • $227.85k - $423.15k

     ...are celebrated, here you can thrive. Principal Research Scientist (Recommender Systems) The Job The...  ...business. The Daily Develop innovative data science solutions that utilize...  ...statistical modeling expertise, such as causal inference modeling, ensembles, neural networks... 
    Principal
    Full time
    Temporary work
    Work experience placement
    Local area

    Warner Bros.

    Bellevue, WA
    8 days ago
  • $229k - $343k

     ...Saturn, and other digital services.The Data Science and Insights team seeks to...  ...data.We’re looking for a Lead Data Scientist to join Snap Inc.What you’ll doAct as...  ...subject matter expert for key experimentation and causal inference methodologiesLead medium to large projects... 
    Full time
    Live in
    Work at office
    Local area

    Snap

    Seattle, WA
    3 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Principal Data Scientist - Experimentation & Causal Inference. Be the first to apply!