Validity and Item Analysis Chapter Validity ∗ Concerns what instrument measures and how well it does so ∗ Not something instrument “has” or “does not have” ∗ Informs counselor when it is appropriate to use instrument and what can be inferred from results ∗ Reliability is a prerequisite for validity Traditional Categories of Validity ∗ Content-Related ∗ Criterion-Related ∗ Construct Validation Evidence: Standards for Educational and Psychological Testing ∗ Evidence Based on Test Content ∗ Evidence Based on Response Processes ∗ Evidence Based on Internal Structure ∗ Evidence Based on Relations to Other Variables ∗ Evidence Based on Consequences of Testing Evidence Based on Test Content ∗ Degree to which the evidence indicates that items, questions, or tasks adequately represent intended behavior domain ∗ Central focus is typically on how the instrument’s content was determined ∗ Content-related validation evidence should not be confused with face validity Evidence Based on Response Processes ∗ Concerns whether individuals respond in a manner consistent with construct measured ∗ May also examine information processing differences by subgroup ∗ Computer simulated performances Evidence Based on Internal Structure ∗ Examining internal structure using factor analysis ∗ Can also examine internal structure of instrument for different subgroups (differential item functioning) Evidence Based on Relations to Other Variables ∗ Correlational method ∗ Convergent and discriminant evidence ∗ Multitrait-multimethod matrix ∗ Prediction or instrument-criterion relationship ∗ Concurrent and predictive validity ∗ Regression ∗ Decision Theory ∗ Validity Generalization Correlational Method ∗ Correlational method  examines relationship between instrument and criterion or other pertinent variables ∗ Validity coefficient ∗ Multitrait-multimethod matrix  examines relationships between instrument and traits that are theoretically related and unrelated to it Prediction/Instrument-Criterion Relationship ∗ Concurrent validity ∗ Predictive validity Regression a = y-intercept b = slope X = score Standard Error of Estimate ∗ The margin of expected error in the individual’s predicted criterion score as a result of imperfect validity Sest = Sy 1− r xy Decision Theory: Expectancy Table Expectancy Table (cont.) Expectancy Table (cont.) Validity Generalization ∗ Method of combining validation studies to determine if validity evidence can be generalized ∗ Must have a substantial number of studies ∗ Must use meta-analytic procedures Evidence-Based Consequences of Testing ∗ Examples of social consequences: ∗ Group differences on tests used for employment selection ∗ Group differences in placement in special education ∗ Counselors should consider both validation evidence and social implications Conclusion on Validation Evidence ∗ Gradual accumulation of evidence ∗ Counselor must evaluate the information to determine if it appropriate for which client under what circumstance ∗ Validation evidence should also be considered in informal assessments Item Analysis ∗ Examining and evaluating each item in instrument ∗ Item difficulty: p = # who answered item correctly total # of individuals ∗ Item discrimination: ∗ Extreme group method: d = upper % – lower % ∗ Correlational method Item Response Theory ∗ Focus is on each item and establishing items that measure ability or level of a latent trait ∗ Involves examining item characteristic curve and calibration of each individual item ∗ Parameters: ∗ Difficulty ∗ Slope (discrimination) ∗ Probability of guessing Item Characteristic Curve [...]... considered in informal assessments Item Analysis ∗ Examining and evaluating each item in instrument ∗ Item difficulty: p = # who answered item correctly total # of individuals ∗ Item discrimination: ∗ Extreme group method: d = upper % – lower % ∗ Correlational method Item Response Theory ∗ Focus is on each item and establishing items that measure ability or level of a latent trait ∗ Involves examining item...Regression a = y-intercept b = slope X = score Standard Error of Estimate ∗ The margin of expected error in the individual’s predicted criterion score as a result of imperfect validity 2 Sest = Sy 1− r xy Decision Theory: Expectancy Table Expectancy Table (cont.) Expectancy Table (cont.) Validity Generalization ∗ Method of combining validation studies to determine if validity evidence can... Evidence-Based Consequences of Testing ∗ Examples of social consequences: ∗ Group differences on tests used for employment selection ∗ Group differences in placement in special education ∗ Counselors should consider both validation evidence and social implications Conclusion on Validation Evidence ∗ Gradual accumulation of evidence ∗ Counselor must evaluate the information to determine if it appropriate for... on each item and establishing items that measure ability or level of a latent trait ∗ Involves examining item characteristic curve and calibration of each individual item ∗ Parameters: ∗ Difficulty ∗ Slope (discrimination) ∗ Probability of guessing Item Characteristic Curve ... Internal Structure ∗ Examining internal structure using factor analysis ∗ Can also examine internal structure of instrument for different subgroups (differential item functioning) Evidence Based on... evaluate the information to determine if it appropriate for which client under what circumstance ∗ Validation evidence should also be considered in informal assessments Item Analysis ∗ Examining and... Concerns what instrument measures and how well it does so ∗ Not something instrument “has” or “does not have” ∗ Informs counselor when it is appropriate to use instrument and what can be inferred