Senior Psychometrician

Role Description

The Senior Psychometrician plays a vital role by conducting timely psychometric analyses across the entire certification product portfolio, leveraging psychological and industry best practices while adhering to ISO 17024 standards. The analysis and reporting deliverables provide critical inputs to the entire high-stakes development process, program security posture, and overall validity and reliability of exam forms and all exam results. The role will need to communicate well, prioritize tasks to ensure deadlines are met, think outside the box, provide innovative solutions to complex problems, including new types of analyses, and participate in the development of technology solutions that will automate basic psychometric analyses. This person will collaborate with Exam Services, IT, Certification Operations and Security teams.

Essential Duties and Responsibilities

• Responsible for the analysis of all exam results and the defensible construction of all exam forms and/or exams delivered leveraging CAT methodologies.
• Identify fraudulent activities via data forensic analyses.
• Work with the Exam Services team members to confirm clear timelines for all analyses needed to launch new and updated certification products.
• Produce timely JTA survey analyses and JTA reports, establishing draft blueprints for Governance Committee (GC) review and approval.
• Conduct item and simulation analysis leveraging a combination of the most appropriate methodologies for the given product, based on available data and project needs:
  • Classical Test Theory (CTT)
  • Item Response Theory (IRT)
  • Rasch
• Conduct analyses in support of standard setting, using SME Angoff ratings and providing Hofstee analyses during standard setting workshops.
• Develop standard setting reports that help inform the cut score selection process and GC committee approval.
• Conduct beta exam rescores leveraging relevant software tools and produce critical outcomes for comparison (audit) against test delivery provider results.
• Build multiple equated forms (based on project criteria) and coordinate clear deliverables and reports to the relevant Exam Services team members to inform exam publishing.
• Generate scaled scoring tables to manage score equating in a confidential manner.
• Generate reports in support of ISO 17024 standards.
• Conduct various routine analyses across the entire portfolio based on quarterly volume, identify potential issues, and recommend solutions or process adjustments based on results:
  • General item statistics (exam health check), including distractor analysis, forms analysis, etc.
  • DPF analysis
  • DIF analysis
  • SSI analysis
  • RSI analysis
  • Others
• Conduct proctoring modality comparisons across ISO 17024 accredited products, ensuring score validity and equivalent meaning of scores in both modalities.
• Conduct large-scale analyses mid-lifecycle to produce larger numbers of forms in support of frequent forms publishing and ongoing product maintenance.
• Conduct special data forensic analyses towards the invalidation of scores when applicable, protecting the brand while ensuring the validity of the overall program.
• Conduct analysis of test center performance, producing z-score reports in support of test center policies and audit practices.
• Conduct ad hoc or custom analyses as needed to identify problems and/or review fairness and flag potential bias or other issues that might be cause for concern.
• Conduct analyses of translated exam forms, performing comparisons and making recommendations as needed.
• Consult with the Exam Services team as needed to set up special research projects, and provide other consultative inputs to development processes, reports, etc.
• Consult with Innovation team regarding the validity of emerging assessment types, assessment methods, learning offerings and new measurement systems/platforms.

Key Performance Indicators

• Conduct analyses and produce reports consistently on time in relation to core certification development project timelines.
• Provide clear, documented input and new requirements for platform certification delivery enhancements and/or certification database enhancements, leveraging automated statistical analysis.
• Conduct scheduled data forensic analyses, providing recommendations for detection and remediation.
• Engage during ANSI audits to address auditor’s psychometric questions to ensure CompTIA certification programs remain accredited.
• Remain abreast of new industry trends, new statistical analyses and considers implementation of new methods to improve validity and reliability throughout the assessment process.

Additional Expectations

• Employees are expected to model CompTIA's values and reflect them in how they work and collaborate.
• Participate in required and role relevant training to maintain skills, ensure compliance, and support ongoing professional development.
• May be required to travel based on role responsibilities and business needs.

Qualifications

• Proficiency with a combination of preferred software tools (e.g. SPSS, R, SAS, Qualtrics, Tucker Equating, IRTPro, flexMIRT, etc) in support of all required analyses conducted throughout the development lifecycle.
• Proficient with data visualization tools (e.g. Tableau, ggplot2, etc).
• Excellent written and verbal communication skills.
• Attention to detail and accuracy in all work.
• Project management skills with the ability to manage multiple tasks, priorities, and deadlines.
• Analytical thinking and complex problem-solving skills.
• Ability to innovate and think creatively to build and deploy new processes and solutions.
• Advanced understanding of certification exam delivery platforms and various methods of delivery (e.g., fixed forms, LOFT, CAT, etc).
• Experience participating in software development projects is a plus.
• Willingness to participate at industry conferences, presenting best practices and new methodologies, maintaining CompTIA’s leadership in the certification industry.
• Effectively communicate complex statistical concepts and results to non-technical stakeholders.
• Demonstrate ethical judgment and integrity when using, manipulating and storing sensitive data, PII and other confidential information.

Education and/or Experience

• Ph.D. in Psychometrics, Educational Measurement, Statistics or a related field.
• 10+ years of direct experience in related field.

Benefits

• Health, Dental, and Vision Insurance & FSA/HSA Plans.
• Performance bonus up to 20% of base salary.
• Unlimited PTO & 15 Paid Holidays.
• Flexible Schedules & Summer Hours.
• 12 weeks of Paid Parental Leave.
• Sponsored Costco or Sam’s Membership.
• 401K Retirement Plan with 6% company match.
• Spot Bonuses for going above & beyond.
• Tuition Reimbursement.
• Home Office Allowance.
• Wellness Reimbursement.
• Student Loan Repayment.
• Broadband Stipend.
• Expected compensation based on experience and qualifications – $200,000 - $225,000.