Subject Matter Expert - Assessment/Psychometrics Focus

Job Description

Job Description

Subject Matter Expert - Assessment/Psychometrics Focus

Position Overview

The Subject Matter Expert will provide psychometric expertise for the design, analysis, validation, and improvement of FEMA training assessments. This individual will ensure test banks, pre- and post-course assessments, capstone evaluations, and performance-based assessment tools are valid, reliable, measurable, and aligned with approved learning objectives and operational requirements.

Key Responsibilities

• Lead the design, review, and validation of assessment items and test banks.
• Conduct content-domain analysis to determine appropriate assessment coverage.
• Develop and evaluate pre-course tests, post-course tests, final examinations, capstone assessments, and performance-based assessments.
• Perform Classical Test Theory item analysis and evaluate item difficulty, discrimination, distractor effectiveness, and overall test performance.
• Assess test reliability and validity and recommend corrective actions when assessments do not meet established standards.
• Support standard-setting activities and establish defensible cut scores for professional training assessments.
• Ensure assessment questions align with Terminal Learning Objectives, Enabling Learning Objectives, course content, competencies, behaviors, and Position Task Book requirements.
• Develop cognitively appropriate, scenario-based questions with effective distractors and clear feedback.
• Prepare answer keys explaining why each answer is correct or incorrect.
• Develop objective scoring guides, evaluation guides, and weighted rubrics for learning activities, tests, and capstone performance.
• Review assessment results and learner-performance data to identify gaps, trends, and opportunities for course improvement.
• Collaborate with instructional designers, training managers, FEMA stakeholders, and emergency-management SMEs.
• Document assessment-validation findings, recommended revisions, actions taken, and final disposition.
• Ensure final assessments meet FEMA/NDEMU quality standards, including required question types, content coverage, and alignment.

Qualifications/Experience

Minimum Qualifications

• Master’s degree in Psychometrics, Educational Measurement, Quantitative Psychology, Industrial/Organizational Psychology, or a related field.
• At least five years of professional experience in content-domain analysis, test development, item writing, and standard setting.
• Demonstrated experience conducting Classical Test Theory item analysis.
• Proven experience evaluating test reliability and validity and establishing assessment cut scores.

Preferred Qualifications

• Ph.D. or Ed.D. in Educational Measurement, Evaluation, Psychometrics, or a closely related discipline.
• Eight or more years of experience developing and validating domain-specific certification examinations or high-stakes assessments for Government programs.

Required Skills

• Psychometric and statistical analysis
• Classical Test Theory
• Test reliability and validity evaluation
• Item writing and test-bank development
• Standard setting and cut-score development
• Learning-objective and assessment alignment
• Rubric and scoring-guide development
• Quantitative data interpretation
• Assessment quality assurance
• Technical documentation
• Collaboration with instructional designers and domain SMEs

Certification

No specific professional certification is identified as mandatory.

Clearance/Security

• Must successfully complete a FEMA public trust background investigation.
• Must complete required FEMA/DHS privacy, records management, insider-threat awareness, and cybersecurity-awareness training.
• Must safeguard unclassified but security-sensitive information.

Place of Performance

Primarily at the contractor’s facilities or virtually. Work may also be performed at FEMA Headquarters or NDEMU in Emmitsburg, Maryland, as required. Contract travel is not required.