Staff Quality Engineer

Job Description

Job Description

The Staff Quality Engineer owns quality for the Unboxed platform. This is a senior individual contributor role with technical authority over how quality is defined, measured, and enforced across every surface we ship, including work produced by teams outside QA. Quality spans four surfaces: a robust learning management system, AI-generated roleplay and practice, coaching workflows, and client-facing analytics and reporting.

The role is hands-on. The Staff Quality Engineer designs the test architecture, writes and reviews automation, runs exploratory testing on high-risk changes, and diagnoses defects personally. The difference between this role and a senior testing role is reach. This person sets standards that other engineers follow, changes how the platform is built so it is testable and holds release authority on behalf of the company.

A significant portion of the role is quality for AI-generated output. Unboxed delivers AI scoring, feedback, and roleplay to enterprise clients in regulated industries. Those clients will ask how we validate that the system performs accurately and fairly across their populations. The Staff Quality Engineer is an owner of the program that produces that answer.

The role includes direct managing, mentoring, and developing two QA Engineers and setting the quality standards that contract engineering and QA partners operate within.

Key Responsibilities

Quality Architecture and Strategy

• Own the test architecture for the platform, including framework selection, automation strategy, test data management, and environment strategy, and make the build-versus-buy decisions that go with it.
• Set the quality standard for the organization: what gets tested, at what layer, to what depth, and what coverage gaps we consciously accept.
• Author and maintain the defect severity rubric, definitions of done, and test conventions used across in-house and contract teams.
• Stay hands-on in the highest-risk areas of the product, including exploratory testing, complex defect diagnosis, and automation for critical paths.

Release Authority

• Hold go/no-go authority for releases against a published, evidence-based standard, and communicate the basis for each decision.
• Own release verification and the quality signal that supports it, so ship decisions rest on data rather than confidence.
• Escalate accepted risk explicitly, with the exposure named, rather than allowing it to pass unrecorded.

AI Quality and Fairness

• Own how quality is measured for AI-generated scoring, feedback, and roleplay, using evaluation approaches built for output variability rather than exact-match expectations.
• Build and maintain the evaluation harness: golden sets, rubric-based scoring, regression gates on model, prompt, and scoring-logic changes, and drift monitoring in production.
• Own the fairness and bias testing program for AI scoring. Run subgroup analysis, monitor for differential performance across learner populations, interpret results, and escalate findings. Partner with measurement and psychometrics expertise on methodology and thresholds.
• Produce the evidence artifacts enterprise clients and their compliance functions rely on to trust AI-generated output and translate client trust concerns into acceptance criteria the team can verify.

Quality Outcomes and Reporting

• Own the outcome metrics for platform quality, including Defect Escape Rate, Critical Path Coverage, and MTTD/MTTR, and report on them directly to leadership on a regular cadence.
• Use those metrics to drive change in how software is built, not only to describe how testing performed.
• Provide leadership with a clear read on quality trends, risk concentration, and where investment is warranted.

Engineering Influence

• Change how engineering builds, so quality moves upstream. Drive testability into design, acceptance criteria into stories, and automated checks into CI.
• Review designs and pull requests for quality risk and raise the bar through direct technical influence rather than process enforcement.
• Earn adoption of quality practices across teams that do not report to this role.

Sourcing and Partner Strategy

• Set the strategy for how testing capacity is sourced, including what stays in-house, what goes to contract partners, and where automation replaces manual effort.
• Define the standards, conventions, and acceptance criteria contract engineering and QA teams operate within, and review their output against the same bar as in-house work.
• Serve as the single point of quality accountability across in-house and contract contributions

Mentorship and Development

• Manage, mentor, and develop two QA Engineers, owning their prioritization, technical growth, and performance with a coaching-first approach.
• Grow each engineer’s ability to reason about quality independently, so team judgment scales without requiring this role in every decision.
• Raise the quality skill level of the broader engineering organization through review, pairing, and documented practice.

Key Skills

• Deep hands-on quality engineering skill: test design, exploratory testing, regression strategy, and defect diagnosis in B2B SaaS. This remains the primary skill the role is hired for.
• Test architecture judgment: the ability to design an automation strategy that holds up over years and to know where automation pays off versus where it becomes maintenance burden.
• Practical experience testing AI/ML or otherwise non-deterministic systems, including evaluation of scored output where there is no single correct answer.
• Statistical literacy sufficient to read a subgroup analysis, distinguish signal from noise, and interpret fairness results responsibly.
• Demonstrated influence without authority: the ability to change how other engineers work through credibility and evidence.
• Fluency with quality metrics and the ability to use them to drive decisions, not just report them.
• Ability to communicate quality risk credibly to technical and non-technical audiences, including enterprise clients.

Requirements

• 8+ years of quality or software engineering experience in B2B SaaS, with demonstrated ownership of quality outcomes rather than test execution alone.
• A track record of staying hands-on at a senior level, with systems or practices they designed that outlived their direct involvement.
• Evidence of organization-level impact: a quality practice, framework, or standard they introduced that changed how a company-built software.
• Experience testing AI-driven, ML-driven, or data-driven features, or clear demonstrated ability to reason about quality under output variability.
• Experience mentoring engineers and setting standards for contract, offshore, or vendor teams.
• Strong automation and CI/CD background with modern testing frameworks.
• Familiarity with enterprise quality expectations, including regulated or compliance-sensitive clients.
• Experience with Agile/Scrum methodologies and tools such as JIRA.
• Bachelor’s degree in a relevant field, or equivalent practical experience.

Benefits

Unboxed team members benefit from our comprehensive compensation and rewards program which includes:

• Competitive salary and benefits
• Ample paid time off
• Dynamic and convenient office location – unlimited snacks, casual dress code, covered parking and gym on site
• Open communication and a commitment to fostering teamwork across the organization

Salary Range: $115,000 - $130,000 annually