Experimentation Analytics — Lead

Role: Experimentation Analytics - Lead

Location: San Francisco, CA (Onsite)

Type: Contract

Position Overview

• We are seeking a Lead Experimentation Analyst to support the growth of our rapid experimentation program and e-commerce experimentation roadmap. This role will serve as a practitioner and subject matter expert in website experimentation, partnering closely with Product, UX, Analytics, and Engineering teams to design experiments, evaluate results, and ensure product decisions are supported by statistically sound evidence.
• The ideal candidate will have strong statistical expertise, hands-on experience running digital experiments at scale, and the ability to translate experimental results into clear, actionable business decisions.

Key Responsibilities:

• Partner with Product and UX teams to develop hypotheses, success metrics, and experimentation strategies for digital experiences.
• Design and execute A/B and multivariate experiments, including audience allocation, sample sizing, test duration, primary KPIs, and guardrail metrics.
• Conduct statistical power analysis and determine minimum detectable effects for proposed experiments.
• Analyze experiment results and assess both statistical and practical significance.
• Apply advanced experimentation techniques, including variance reduction, sequential testing, heterogeneous treatment effect analysis, and other appropriate statistical methodologies.
• Evaluate experiment validity by identifying sample ratio mismatch, instrumentation issues, novelty effects, traffic imbalance, and other potential sources of bias.
• Develop measurement approaches for experiments involving uneven audience allocation or operational constraints.
• Conduct segmentation and follow-up analyses to understand how experiment effects vary across devices, customer groups, product categories, and digital experiences.
• Establish experimentation standards, governance, and best practices across digital product teams.
• Partner with analytics implementation teams to validate tracking and ensure experiment data quality.
• Translate experimental findings into clear launch, iterate, or stop recommendations for Product teams and senior stakeholders.
• Support the continued development and maturity of the organization's experimentation and measurement practices.

Required Qualifications:

• Bachelor's degree or equivalent practical experience.
• 4 6 years of experience in experimentation analytics, digital analytics, data science, statistics, or a related domain.
• Advanced proficiency with domain-specific experimentation and analytics tools.
• Strong statistical knowledge and hands-on experience with digital experimentation.
• Strong business acumen with Retail industry experience, preferably within e-commerce or digital product environments.
• Excellent verbal and written communication skills, with the ability to present technical findings to non-technical business partners and executive-level stakeholders.
• Strong attention to detail and organizational skills, with the ability to manage multiple work requests, projects, and initiatives simultaneously.
• Experience managing or guiding agency/vendor work.
• 1 2 years of experience serving as a team lead or peer lead, including coaching and mentoring junior team members, is preferred.