Skip to main content

Developing Your Assessment Strategy

Updated

Background

The following information was presented as part of a four part OPM-led series for agencies on how to develop their assessment strategy. The content was discussed during reoccurring USA Hire Workgroup Meetings by a lead I/O Psychologist for the OPM HR Center for Assessment and Evaluation.

The high-level outline of the information discussed has been consolidated and shared here for reference.

 

Table of Contents

Use this table to contents to jump to a particular section of interest.

 

PART 1: QUESTIONS AND CONSIDERATIONS

DEFINING ASSESSMENT

  • Assessment

    • A systematic approach to gathering information about individuals
    • Used to make selection, promotion, or other career-related decisions

  • Assessment Tool

    • Any procedure used to measure an individual’s employment or career-related qualifications and interests
    • Includes objective, subjective, and projective measures

  • Assessment Strategy

    • Plan for designing and implementing one or more assessment tools for an organization, occupation, or a specific situation

  • Reliability

    • How well an assessment measures a characteristic
    • Produces dependable, repeatable, and consistent information
    • Reliability ranges from 0 to 1
    • What is a “good” reliability coefficient?

  • Validity

    • What is measured and how well it is measured
    • Degree to which an assessment accurately predicts an outcome
    • Validity ranges from 0 to 1
    • What is a “good” validity coefficient?

CONTEXT

Assmt Strt 1.png

The Federal Landscape

  • Delegated examining
  • Merit promotion
  • Direct hire authority
  • The time vs quality conundrum

Barriers to Effective Assessment

  • Minimal assessment knowledge/expertise
  • Lack of collaboration among key players
  • Constraints on time and/or resources
  • Misperception/confusion on policies and procedures
  • Availability of meaningful and/or useful data to support assessment development
  • System limitations or constraints

"Simple" Context Questions

  • What process was used previously and was it effective?
  • How many job openings are expected?
  • How many applicants are expected per opening?
  • Is the job general or specialized in nature?
  • What is the level of the job?
  • Is this a hard to fill position, and if so, why?
  • What are the characteristics of the applicant pool?
  • What resources (people, time money) are available?

"Complex" Context Questions

  •  What other factors do we need to consider in filling this position?
    • Satisfaction with previous applicants and hires
    • Cost of a hiring error
    • Union involvement
    • Litigation history / potential
    • Turnover / retention
  • Does your agency support the use of assessments?
    • Advocates
    • Detractors

 

STAKEHOLDERS

Assmt Strt 2.png

Primary Stakeholders

  • Agency leadership
  • Hiring Managers
  • HR Personnel
  • Job Applicants

Secondary Stakeholders

  • Subject Matter Experts
  • Personnel Psychologists
  • Information Technology Staff
  • Tax Payers

 

 

CONTENT

Assmt Strt 3.png

Content Questions

  • What is the intended purpose of the assessment?
  • Has a job analysis been conducted?
  • Has the job changed significantly since the last job analysis?
  • What are the critical competencies?
  • What competencies are required at entry?
  • What competencies do you want to measure and why?
  • How is success measured on the job?
  • How will success be measured for the assessment?

 

 

PROCESS

Assmt Strt 4.png

Process Questions

  • How will the assessments be delivered?
  • What assessment approach will be used?
  • What is the assessment administration order?
  • How will the assessments be scored?
  • Have you evaluated each assessment option for:
    • Reliability and validity?
    • Competencies assessed?
    • Assessment double dipping or dichotomies?
    • Overall value relative to other options?

 

OTHER THINGS TO CONSIDER

  •  Practicality / viability / feasibility of the assessment process
    • Sometimes less is better…

  • What are you really trying to predict?
    • Retention, training success, job performance, hiring manager happiness…

  • And the question that is most frequently forgotten, but may be the most important of all…
    • Do you really need an assessment?

 

PART 2: TYPES OF ASSESSMENTS 

DEFINING GOOD ASSESSMENT

  • Reliability

    • How well an assessment measures a characteristic
    • Produces dependable, repeatable, and consistent information
    • Reliability ranges from 0 to 1
    • Good reliability is .80 to .89, Excellent reliability is .90 and above

  • Validity

    • What is measured and how well it is measured
    • Degree to which an assessment accurately predicts an outcome
    • Validity ranges from 0 to 1
    • Useful assessments have validities in the .20 to .60 range

 

TYPES OF ASSESSMENT

Job Knowledge or Technical Skills Tests
Evaluate job knowledge, technical skills, or professional expertise in job specific competency areas

Cognitive Ability Tests
Assess abilities involved in thinking (e.g., reasoning, memory, verbal and mathematical ability, problem solving)

 

Situational Judgment Tests
Assess decision making and other competencies by presenting applicants with a job-related problem or situation and asking them to identify their most and least likely courses of action

Non-Cognitive and Biographical Assessments
Assess personality attributes, attitudes, experiences, interests, skills, and abilities validated as predictors of overall performance for a given occupation

Work Sample Assessments
Evaluate an applicant’s ability to perform tasks or work activities that mirror the tasks employees perform on the job

Assessment Centers
Assess multiple competencies by requiring applicants to complete a series of exercises that simulate situations, problems and tasks from the job

Writing Assessments
Writing skills tests focus on general aspects such as grammar and communicating information in a succinct and organized manner

Work sample writing tests require applicants to produce writing samples that are similar to those produced on the job

Structured Interview Assessments
Require applicants to respond to a set of standardized questions, linked to a job-relevant competencies, in which rating scales and benchmarks provide consistent guidelines for scoring

 

VALIDITY OF COMMON ASSESSMENT TYPES

 

Assessment Type Validity
Work sample  .54
Cognitive ability .51
Structured Interviews  .51
Job knowledge / technical skills  .48
Assessment centers  .37
Non-cognitive / biographical  .35
Situational judgement  .34
Assessment questionnaires (no longer used as of September 30, 2025) .10 to .30*

 

 

INCREMENTAL VALIDITY

  • “The improvement obtained by adding a particular procedure or technique to an existing combination of assessment methods”   - American Psychological Association

    • Incremental validity reflects the value of combining assessments to meet your hiring needs

  • Two key concepts about incremental validity: 
    1. It is not an additive relationship. Adding a non-cognitive test to a cognitive test results in a validity of about .60, not .86

    2. “Need” is key to understanding the value of incremental validity. If the need is to hire quickly, the incremental validity may be different than if the need is to hire top performers

 

USA HIRE

Example of a Poor Quality Assessment's Results

 

Example of a High Quality Assessment's Results

 

 

USA HIRE CHALLENGES & CONSIDERATIONS

  • Measures general competencies, not job-specific and/or technical competencies
  • Requires first-time applicants to complete a 1-2 hour assessment battery (USA Hire Standard)
  • Cut scores are not available for all series (USA Hire Standard)
  • Potentially less suitable for hard to fill positions, low volume JOAs, and highly specialized positions
  • Not recommended as the only assessment tool

 

USA HIRE RECOMMENDED BEST PRACTICES

 

 

PART 3: THE WHOLE PERSON APPROACH

WHOLE PERSON APPROACH

Measure as much of the whole person as practically possible

 

 

 

WHOLE PERSON APPROACH & JOB PERFORMANCE

 

 

WHOLE PERSON JOB ANALYSIS

  • Consider the broad domain of the job
  • Identify as many of the key competencies making up this broad domain

 

BENEFITS OF A WHOLE PERSON JOB ANALYSIS

  • Allows you to consider the full competency range
  • Allows you to make better assessment decisions
  • Provides insight into your weighting strategy
  • Results in a more diverse pool of applicants
  • Results in higher validity (better job performance)
  • Increases the ROI of your hiring process

 

CONDUCTING A WHOLE PERSON JOB ANALYSIS

  • Ask questions about competency categories
  • If you’re using USA Hire, general competencies have been covered
    • Focus on other key aspects of the job
      • Technical competencies
      • Job specific competencies
      • Job knowledge
      • Communication
  • Not sure where to start, call your friendly personnel
    research psychologist!

 

ILLUSTRATION OF A WHOLE PERSON ASSESSMENT

To accurately assess the whole person, you need to use a palette of assessments

Whole Person Assmt.png

 

BUILDING WHOLE PERSON ASSESSMENTS

 

 

GOAL OF A WHOLE PERSON ASSESSMENT

 

  • We are not trying to measure everything
  • The goal is to complete enough of the puzzle to make a sound and effective decision

 

 

PART 4: ASSESSMENT APPROACHES FOR COMMON HIRING CONSIDERATIONS

A NOTE ABOUT THE CONTENT BELOW

  • Addresses common (and sometimes not so common) hiring situations and potential assessment strategies for these situations
  • Focus is on providing examples of assessments readily available and/or often used in the Federal
    government
  • Your assessment strategy should drive assessment choice
  • When in doubt, ask your friendly Personnel (Research) Psychologist for assistance!

 

HIRING AUTHORITIES

Delegated Examining

  • In a typical Delegated Examining scenario, we know little about the applicants
    • A palette of assessments is usually the best strategy
    • USA Hire belongs in your palette

  • Entry-level, no technical competencies
    •  USA Hire Standard, Early Career Talent Assessment, and/or Automated Writing Assessment

  • Mid-level, some technical competencies required
    • USA Hire Standard or Premium, Resume, Structured Interview, Assessment to measure technical skills

  • Senior-level, technical competencies differentiate
    • USA Hire Premium or Custom, SME-Based Process, Structured Interview, Assessment to measure technical skills

Merit Promotion - Government Wide

  • Same recommendations listed for Delegated examining above.

Merit Promotion - Internal

  • Internal merit promotion comes in two broad flavors:
    • Limited to a specific group (e.g., department, division)
    • Open to all within the agency

  • Limited Merit Promotion or smaller agencies
    • SME-Based Process, Structured Interview, Assessment to measure technical skills
    •  USA Hire may be beneficial if general competencies differentiate

  • Agency-wide Merit Promotion for large agencies
    • Same recommendations listed for Delegated examining above.

 

Direct Hire

  • Myth: Assessments cannot be used with Direct Hire Authorities (DHAs)
    • Truth: Rarely do DHAs limit the use of assessments. You will need an assessment strategy for most direct hire scenarios.

  • When using DH to speed up the hiring process
    • USA Hire Standard or Premium with a secondary assessment to measure technical skills can facilitate the process
    • When minimum proficiency is critical, cut scores can be effective

  • When using DH to address hard to fill, specialized roles
    • SME-Based Process, Structured Interview, Assessment to measure technical skills

 

Pathways Program

  • Student Intern, no technical competencies required
    • Early Career Talent Assessment (xx99), WSA, Structured Interview

  • Student Intern, some technical competencies required
    • Early Career Talent Assessment (xx99), SME Process, Structured Interview

  • Recent graduate, no technical competencies required
    • USA Hire Standard (series specific), WSA, Structured Interview

  • Recent graduate, some technical competencies required
    • USA Hire Standard (series specific), SME-Based process, Structured Interview, Assessment to measure technical skills

  • Recent graduate, technical competencies differentiate
    • USA Hire Premium or Custom, SME-Based Process, Structured Interview, Assessment to measure technical skills

 

Veterans Preference and ICTAP/CTAP

  • Myth: Assessments add no to little value when veterans preference, ICTAP/CTAP, or other preference-based hiring approaches are in place
    • Truth: When used appropriately, assessments help to ensure a qualified individual is hired under preference-based approaches

  • Preference applies to applicants who are qualified
    •  A combination of assessments measuring the required general competencies (USA Hire) and technical competencies (SME Process, Structured Interview, other assessment alternative) help identify qualified applicants
    • Cut scores can be used to ensure minimum level of proficiency

 

OTHER HIRING SITUATIONS

Supervisory Positions

  • Making an effective supervisory or managerial level hire is critical to the long-term success of the organization

  • A common mistake is hiring for technical competence, rather than leadership competence

  • USA Hire offers 3 selection-focused leadership assessments
    • Federal Supervisor Assessment (off-the-shelf)
    • Supervisory Situational Judgement Test (tailored)
    • Executive Assessment (off-the-shelf)

  • One effective assessment strategy is to use a USA Hire leadership assessment, a secondary assessment to measure technical skills, and a structured interview

 

High Applicant Volume

  • A well designed and well implemented assessment strategy can be your best friend in high volume applicant situations
    • Goal is to successfully manage the volume while identifying top applicants

  • Well suited for multiple hurdles
    • Future USA Staffing capability

  • Alternative assessment to address eligibility and technical aspects 

  • USA Hire is an effect tool
    • Reduces casual appliers
    • Cut scores ensure minimum proficiency
    • Effectively differentiates among applicants

 

Low Applicant Volume

  • Myth:  Assessments should not be used in low applicant volume situations
    • Truth: Assessments may be even more important in low applicant situations

  • A sound assessment strategy is critical
    • Avoid “panic” or “warm body” hiring
    • Critical competencies are still required for job success
    • Resume review, SME-Based Process, Structured Interview

Hard to Fill Positions

  • Same recommendations listed for Low Applicant volume above.

 

Return to Main Page

Was this article helpful?

0 out of 0 found this helpful
Return to top