Measurement Validity: A Shared Standard for Qualitative and Quantitative Research
Measurement Validity: A Shared Standard for Qualitative and Quantitative Research (DOI: https://doi.org/10.1017/S0003055401003100) was written by Robert Adcock and David Collier in 2001. It was published in American Political Science Review (vol. 95, no. 3).
The authors describe measurement validity, which they fit into the King, Keohane, and Verba framework as a component of descriptive validity. The authors argue that this framework for measurement validity can put qualitative methods on par with quantitative ones.
"Valid measurement is achieved when scores (including the results of qualitative classification) meaningfully capture the ideas contained in the corresponding concept."
Compare measurement validity to measurement error. A systemic measurement error is a bias. A random measurement error is a problem for reliability. Measurement validity is concerned with biases (among other problems), but not with reliability.
The recommended procedure to establish content validity (i.e., does a score include irrelevant content? does a score exclude relevant content?) is to audit the component measurements by comparing the core concept to what is actually captured. For example, Paxton (2000) noted that a common concept in democracy indexes is universal suffrage, but this is often measured in such a way that male suffrage is actually indicated.
There is also a case-oriented procedure which involves evaluating whether inclusion of a measure is 'sorting' cases in the right directions. For example, O'Donnell (1996) noted that a concept in democratic consolidation is capacity to withstand crises. By this measure however, several Latin American democracies are more consolidated than southern European ones. O'Donnell rejects this as absurd.
The recommended approach to establish convergent validity (or discriminant validity) is to test the correlation of component measures. If they do not correlate well, then the index is necessarily unstable. For example, Bollen (1980) examined voter turnout and found it did not correlate with other proposed measures of democracy. This divergence suggests that political participation is distinct from political democracy.
The recommended approach to establish nomological validity (or construct validity, but not what most psychometrists refer to by that term) is to test an established relation using the index. For example, to demonstrate that Japan experienced feudalism, Anderson (1974) investigated the degree to which the socioeconomic trends caused by feudalism in Europe were also found in Japanese history. "The basic parallelism of the two great experiences of feudalism, at the opposite ends of Eurasia, was ultimately to recieve its most arresting confirmation of all, in the posterior destiny of each region." (p.414).