Senior QA Automation Engineering

Senior QA Automation Engineer

College Board - Technology - AI Services Infrastructure

Location: This is a fully remote role. Candidates who live near CB offices have the option of being fully remote or hybrid (Tuesday and Wednesday in office).

Type: This is a full-time position

We're a small, close-knit team enabling AI services for specific departments within College Board, working at the intersection of Product, Engineering, and Psychometrics. You'll help build and evolve software that supports College Board's large-scale assessment programs, ensuring accurate, reliable outcomes for the students and educators who rely on them. We're currently building and evolving this capability internally, with a focus on improving efficiency, reliability, and the day-to-day experience of the students and large educator workforce involved in this process. Being small means everyone has real influence: you'll wear multiple hats, partner closely with Product and Psychometrics, and help shape a broader ecosystem of scoring and orchestration services as it takes shape.

As a Senior QA Automation Engineer, you will be helping to develop and validate critical components that will enhance learning opportunities and feedback for millions of students, while also contributing to the needs of educators. For this team, you will be the primary quality-focused engineer, ensuring that solutions coming out of the team are well-crafted and meet the needs of all stakeholders from both a functional and technical standpoint. You will also be asked to help explore new technologies, identify opportunities that benefit Technology as a whole, and participate in AI-driven SDLC processes.

Design & Implementation (60%)

Design and implement testing strategies, including agentic AI, to provide automated validation of quality metrics and gates within the system. Identify appropriate testing strategies to apply for differing scenarios and processes. Provide expert guidance on functional and data-oriented risks.

Help ensure that system components are built with security as a high priority.

Maintain the highest engineering standards in collaboration with product owners, architects, and stakeholders, in support of our systems and services

Understand new product capabilities and decompose the implementation into specific functional changes for verification

Provide entry and exit points for automated services that other engineers may use to ensure their own quality before it reaches the team level.

Build solutions using the best practices of Agile, CI/CD, and DevOps.

Team Operations & Success (20%)

Participate in and possibly lead Agile ceremonies. Participate and lead peer reviews of software engineering artifacts for both design and code

Assist in the resolution of production issues

Be a fierce advocate for development standards and security policies and procedures

Adopt the continuous integration and continuous delivery (CI/CD) practice of rapidly implementing, testing, and delivering high-quality code based on the team’s DevOps model

Continuously develop the skills required to work as part of the Agile team in a poly-skilled development environment

Systems Analysis and Development (20%)

Develop and maintain a thorough understanding of the customer’s business processes and operations

Work closely with Architects and Engineers to evaluate feature requests, provide level-of-effort estimates, and contribute to planning.

Analyze and conduct testing to determine the most efficient approaches for our systems and benchmark current approaches

7+ years of experience in software systems, testing, quality assurance and scripting/development

Working experience with quality tools, such as Cucumber, Jest, Postman, etc.

Familiarity with advanced testing strategies such as using test doubles (stubs, fakes, shims, mocks)

Basic understanding of performance testing and the mechanics of operating one

Experience applying AI for both development and testing purposes

Experience testing code and scripting within an Amazon Web Services environment (Lambda, Step Functions, S3, Dynamo, etc.)

Experience with CI/CD and DevOps, infrastructure and how role access works

Experience working in Agile methodologies

Ability to travel 3-5 times a year to our NYC or Reston, VA office

Curiosity and enthusiasm for emerging technologies, with a willingness to experiment with and adopt new AI-driven solutions and comfort with learning and applying new digital tools independently and proactively.

A drive for impact and excellence: solving complex problems, making data-informed decisions, prioritizing what matters most, and continuously improving through learning, user input, and external benchmarking.

Application review will begin immediately and will continue until the position is filled. This role is expected to accept applications for a minimum of 5 business days. · While the hiring process may vary, it generally includes: resume and application submission, recruiter phone/video screen, hiring manager interview, performance exercise such as live coding, a panel interview, a conversation with leadership and reference checks.

At College Board, we offer more than a paycheck- we provide a meaningful career, a supportive team, and a comprehensive package designed to help you thrive. We’re a self-sustaining nonprofit that believes in fair and competitive compensation grounded in your qualifications, experience, impact, and the market.

We aim to make our best offer upfront, rooted in fairness, transparency, and market data.

LI-REMOTE

#