V
Selection Validation, Differential Prediction, Validity Generalization, and Classification Efficiency
Scott H.Oppler, Rodney A.McCloy, Norman G.Peterson, Teresa L.Russell, and John P.Campbell
This chapter reports results of validation analyses based on the first-tour longitudinal validation (LVI) sample described in Chapter 9. The questions addressed include the following: What are the most valid predictors of performance in the first term of enlistment? Do scores from the Experimental Battery produce incremental validity over that provided by the ASVAB? What is the pattern of predictor validity estimates across the major components of performance? How similar are validity estimates obtained using a predictive versus concurrent validation design? When all the predictor information is used in an optimal fashion to maximize predictive accuracy, what are the upper limits for the validity estimates? This chapter summarizes the results of analyses intended to answer these questions and others related to the prediction of entry-level performance.
THE âBASICâ VALIDATION
This chapter will first describe the validation analyses conducted within each predictor domain. We call these the basic analyses. The final section will focus on maximizing predictive accuracy using all information in one equation.
Procedures
Sample
The results reported in this chapter were based on two different sample editing strategies. The first mirrored the strategy used in evaluating the Project A predictors against first-tour performance in the concurrent validation phase of Project A (McHenry, Hough, Toquam, Hanson, & Ashworth, 1990). To be included in those analyses, soldiers in the CVI sample were required to have complete data for all of the Project A Trial Battery predictor composites, as well as for the ASVAB and each of the CVI first-tour performance factors. Corresponding to this strategy, a validation sample composed solely of individuals having complete data for all the LV Experimental Battery predictors, the ASVAB, and the LVI first-tour performance factors was created for longitudinal validation analyses. This sample is referred to as the âlistwise deletionâ sample.
Table 13.1 shows the number of soldiers across the Batch A MOS who were able to meet the listwise deletion requirements. LVI first-tour performance measures were administered to 6,815 soldiers. Following final editing of the data, a total of 6,458 soldiers had complete data for all of the first-tour performance factors. The validation sample was further reduced because of missing predictor data from the ASVAB and the LV Experimental Battery.
TABLE 13.1
Missing Criterion and Predictor Data for Soldiers Administered LVI First-Tour Performance Measures Number of soldiers: - in the LVI Sample âŠâŠâŠâŠâŠâŠâŠâŠâŠâŠâŠâŠâŠâŠâŠâŠâŠ. 6,815
- who have complete LVI criterion data âŠâŠâŠâŠâŠâŠâŠâŠâŠ.. 6,458
- and who have ASVAB scores âŠâŠâŠâŠâŠâŠâŠâŠâŠâŠâŠâŠâŠ6,319
- and who were administered LV Experimental Battery
(either paper-and-pencil or computer tests) âŠâŠâŠâŠâŠâŠâŠâŠ4,528 - and for whom no predictor data were missing âŠâŠâŠâŠâŠâŠâŠ. 3,163
|
Of the 6,319 soldiers who had complete criterion data and swhose ASVAB scores were accessible, 4,528 were administered at least a portion of the Experimental Battery (either the paper-and-pencil tests, the computer tests, or both). Of these, the total number of soldiers with complete predictor data was 3,163.
The number of soldiers with complete predictor and criterion data in each MOS is reported in Table 13.2 for both the CVI and LVI data sets. With the exception of the 73 soldiers in MOS 19E, the soldiers in the right-hand column of the table form the LVI listwise deletion validation sample. MOS 19E was excluded from the longitudinal validation analyses for three reasons. First, the sample size for this MOS was considerably smaller than that of the other Batch A MOS (the MOS with the next smallest sample had 172 soldiers). Second, at the time of the analyses the MOS was being phased out of operation. Third, the elimination of 19E created greater correspondence between the CVI and LVI samples with respect to the composition of MOS (e.g., the ratio of combat to noncombat MOS).
In the alternative sample editing strategy, a separate validation sample was identified for each set of predictors in the Experimental Battery (see below). More specifically, to be included in the validation sample for a given predictor set, soldiers were required to have complete data for each of the
TABLE 13.2
Soldiers in CVI and LVI Data Sets With Complete Predictor and Criterion Data by MOS MOS | | CVI | LVI (Listwise Deletion Sample) |
11B | Infantryman | 491 | 235 |
13B | Cannon Crewmember | 464 | 553 |
19Ea | M60 Armor Crewmember | 394 | 73 |
19K | M1 Armor Crewmember | â | 446 |
31C | Single Channel Radio Operator | 289 | 172 |
63B | Light-Wheel Vehicle Mechanic | 478 | 406 |
71L | Administrative Specialist | 427 | 252 |
88M | Motor Transport Operator | 507 | 221 |
91A | Medical Specialist | 392 | 535 |
95B | Military Police | 597 | 270 |
Total | | 4,039 | 3,163 |
a MOS 19E not included in validity analyses.
first-tour performance factors, the ASVAB, and the predictor composites in that predictor set only. For example, a soldier who had data for the complete set of ABLE composites (as well as complete ASVAB and criterion data), but was missing data from the AVOICE composites, would have been included in the âsetwise deletionâ sample for estimating the validity of the former test, but not the latter.
There were two reasons for creating these setwise deletion samples. The first reason was to maximize the sample sizes used in estimating the validity of the Experimental Battery predictors. The number of soldiers in each MOS meeting the setwise deletion requirements for each predictor set is reported in Table 13.3. As can be seen, the setwise sample sizes are considerably larger than those associated with the listwise strategy.
The second reason for using the setwise strategy stemmed from the desire to create validation samples that might be more representative of the examinees for whom test scores would be available under operational testing conditions. Under the listwise deletion strategy, soldiers were deleted from the validation sample for missing data from any of the tests included in the Experimental Battery. In many instances, these missing data could be attributed to scores for a given test being set to missing because the examinee failed to pass the random response index for that test, but not for any of the other tests. The advantage of
TABLE 13.3
Soldiers in LVI Setwise Deletion Samples for Validation of Spatial, Computer, JOB, ABLE, and AVOICE Experimental Battery Composites by MOS
the setwise deletion strategy is that none of the examinees removed from the validation sample for a given test were excluded solely for failing the random response index on a different test in the Experimental Battery.
As a final note, there is no reason to expect systematic differences between the results obtained with the listwise and setwise deletion samples. However, because of the greater sample sizes of the setwise deletion samples, as well as the possibly greater similarity between the setwise deletion samples and the future examinee population, it is possible that the validity estimates associated with these samples may be more accurate than those associated with the listwise deletion sample.
Predictors
The predictor scores used in these analyses were derived from the operationally administered ASVAB and the paper-and-pencil and computerized tests administered in the Project A Experimental Battery. For the ASVAB, three types of scores were examined. These scores, listed in Table 13.4, include the nine AS VAB subtest scores (of wh...