Principal Applied Scientist Copilot Quality

Microsoft Copilot is driving the next generation of Copilot experiences used by millions of people every day. As a member of the Copilot organization you will define and drive the strategy technical roadmap and execution required to make Copilot the most trusted capable and continuously improving AI assistant for work and life.

Our mission is make Copilot the best AI for work by turning every meaningful customer signal into lasting product improvement. This means building a quality system that identifies success and failures represents real-world usage determines the highest-value opportunities and verify improvements reach customers safely and reliably with machine learning at its core.

We are seeking an experienced Principal Applied Scientist leader who can elevate our quality systems shape technical and product direction and lead high-impact teams operating across evaluation experimentation production intelligence AI safety reliability customer feedback and product improvement. This role will establish a shared quality bar across Copilot experiences and accelerate the loop from customer signal to diagnosis evaluation improvement and verified product impact.

Microsofts mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset innovate to empower others and collaborate to realize our shared goals. Each day we build on our values of respect integrity and accountability to create a culture of inclusion where everyone can thrive at work and beyond.

Build trusted measures of quality

• Develop online and offline metrics representative evaluation datasets human and AI-assisted labeling approaches and benchmarks grounded in real customer tasks.

• Validate that quality measures are statistically sound sensitive to meaningful changes and predictive of customer outcomes.

Turn signals into improvements

• Identify high-impact failure modes across Copilot experiences and use data evaluations and customer feedback to understand their root causes.

• Improve quality through hill-climbing approaches such as post-training retrieval and ranking prompting orchestration and product changes.

• Design experiments that distinguish promising offline results from improvements customers actually experience.

Experimentation and Release Quality

• Establish rigorous A/B experimentation quality gates regression protection and rollout criteria that connect offline gains with improvements in satisfaction task success engagement retention latency reliability safety and cost.

• Create operating mechanisms that rapidly resolve quality tradeoffs dependencies and release risks.

Required Qualifications:

• Bachelors Degree in Statistics Econometrics Computer Science Electrical or Computer Engineering or related field AND 8 years related experience (e.g. statistics predictive analytics research)
  • OR Masters Degree in Statistics Econometrics Computer Science Electrical or Computer Engineering or related field AND 6 years related experience (e.g. statistics predictive analytics research)
  • OR Doctorate in Statistics Econometrics Computer Science Electrical or Computer Engineering or related field AND 5 years related experience (e.g. statistics predictive analytics research)
  • OR equivalent experience.

Preferred Qualifications:

• Advanced degree (MS or PhD) in a relevant technical discipline.
• Experience improving large language models AI assistants agents or other large-scale AI products.
• Ability to operate in a fastpaced researchdriven environment while maintaining high technical standards.
• Passion for advancing humanaligned AI and staying at the forefront of emerging modeltraining technologies.
• Demonstrated commitment to inclusive leadership talent development and fostering a culture where teams do the best work of their careers.
• Significant experience building evaluating or improving production ML or AI systems.
• Demonstrated expertise in at least one relevant area such as large-model evaluation deep learning post-training reward modeling reinforcement learning retrieval or ranking.
• Experience designing rigorous metrics or experiments and using results to guide technical and product decisions.

#CAP #Copilot

Applied Sciences IC6 - The typical base pay range for this role across the U.S. is USD $165600 - $296400 per year. There is a different range applicable to specific work locations within the San Francisco Bay area and New York City metropolitan area and the base pay range for this role in those locations is USD $220800 - $331200 per year.

Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:
position will be open for a minimum of 5 days with applications accepted on an ongoing basis until the position is filled.

Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age ancestry citizenship color family or medical care leave gender identity or expression genetic information immigration status marital status medical condition national origin physical or mental disability political affiliation protected veteran or military status race ethnicity religion sex (including pregnancy) sexual orientation or any other characteristic protected by applicable local laws regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process read more about requesting accommodations.

Required Experience:

IC