Psychometrician

Role Description

Our client, an education company, is seeking a detail-oriented Psychometrician to lead the design, validation, and analysis of the assessments underpinning its academic programs. This engagement is directly tied to the organization's upcoming re-accreditation review: the Psychometrician will ensure that assessment practices, evidence, and documentation meet accrediting body standards, while also strengthening the validity, reliability, and fairness of assessments for all test-takers.

Key Responsibilities

• Design and develop test items, scales, and assessment instruments aligned with defined constructs and learning/behavioral objectives
• Conduct item analysis, reliability testing, and validity studies (content, construct, criterion)
• Apply Classical Test Theory (CTT) and/or Item Response Theory (IRT) models to evaluate and refine test items
• Perform equating, standard setting, and scaling to ensure comparability across test forms and administrations
• Conduct differential item functioning (DIF) analysis to identify and address potential bias across demographic groups
• Partner with subject matter experts to build and review item banks
• Analyze assessment data and produce technical reports summarizing psychometric properties, score interpretations, and recommendations
• Ensure assessments comply with relevant professional standards (e.g., APA/AERA/NCME Standards for Educational and Psychological Testing) and legal/regulatory requirements
• Compile and organize psychometric evidence, technical documentation, and validity/reliability reports needed to support the re-accreditation self-study and site visit
• Serve as the assessment subject-matter expert in preparing responses to accreditor inquiries and addressing any prior findings or areas of concern
• Support the design of pilot studies and field tests for new or revised instruments
• Collaborate with academic leadership, product, research, and data teams to translate psychometric findings into actionable improvements
• Stay current on advances in measurement theory, testing technology, and accreditation requirements in the education sector

Qualifications

• Master's or Ph.D. in Psychometrics, Educational Measurement, Quantitative Psychology, Statistics, or a related field
• Strong foundation in CTT, IRT, and test validity/reliability frameworks
• Proficiency with statistical software (e.g., R, SPSS, SAS, Python) and psychometric-specific tools (e.g., BILOG, Winsteps, flexMIRT, jMetrik)
• Experience analyzing large-scale assessment data sets
• Excellent written communication skills for producing technical reports for both technical and non-technical audiences

Requirements

• Experience with computerized adaptive testing (CAT) or automated test assembly
• Familiarity with fairness and bias detection methodologies (e.g., DIF, disparate impact analysis)
• Background in credentialing, certification, educational, or workforce assessment contexts
• Prior experience supporting an accreditation self-study, site visit, or accreditor response process
• Experience presenting findings to stakeholders, academic leadership, or accreditation committees