Abstract
Establishing adequacy of psychometric properties of an instrument involves acquisition and evaluation of evidence based on item content and internal structure. Content validity evidence consists of subject matter experts providing quantitative ratings of the extent to which items are a representative sample of targeted domain. Evidence of internal structure includes factor analytic studies and examination of item interrelationships based on item responses from participants. Although subject matter expert ratings and participant response data are traditionally analyzed separately, each serves to inform the other in important ways. We propose integrating subject matter experts’ and participants’ data seamlessly to establish a unified model of validity evidence. The approach is applied to an instrument designed to measure nursing home culture change (i.e., resident-centered care). The proposed method has been demonstrated to be useful with a posterior distribution resulting in stable estimates of psychometric parameters superior to traditional analytic approaches. To illustrate the efficacy of the methodology, we present a simulation study and discuss its place in psychometric methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Albert J.H., Chib S.: Bayesian analysis of binary and polychotomous response data. J. Am. Stat. Assoc. 88, 669–679 (1993)
Bhattacharya J., Choudhry K., Lakdawalla D.: Chronic disease and trends in severe disability in working age populations. Med. Care 46(1), 92–100 (2008)
Cochrane B.B., Lewis F.M.: Partner’s adjustment to breast cancer: a critical analysis of intervention studies. Health Psychol. 24(3), 327–332 (2005)
Cohen, J.: Statistical power analysis for the behavioral sciences. 2nd edn. Lawrence Erlbaum Associates, Hillsdale (1988)
Cooke R.M.: Experts in uncertainty: opinion and subjective probability in science. Oxford University Press, New York (1991)
Crocker L., Algina J.: Introduction to Classical and Modern Test Theory. Harcourt, Brace, Jovanovich, Boston (1986)
Deville C.W.: An empirical link of content and construct validity evidence. Appl. Psychol. Meas. 20, 127–139 (1996)
DeVellis, R.F.: Scale Development: Theory and Applications. 2nd edn. Sage, Thousand Oaks (2003)
Edwards W., Lindman H., Savage L.J.: Bayesian statistical inference for psychological research. Psychol. Rev. 70, 450–499 (1963)
Embretson S.E.: Generating items during testing: psychometric issues and models. Psychometrika 64, 407–433 (1999)
Fagan R.M.: Pioneer network: changing the culture of aging in America. J. Soc. Work Long-Term Care 2, 125–140 (2003)
French S.: Group consensus probability distributions: a critical survey (with discussion). In: Bernardo, J.M., DeGroot, M.H., Lindley, D.V., Smith, A.F.M. (eds) Bayesian Statistics, pp. 183–197. North-Holland, Amsterdam (1985)
Gajewski B.J.: Comments on ‘A note on the power prior’ by Neuenschwander, Branson, Spiegelhalter. Stat. Med. 25(11), 1976–1977 (2010)
Gajewski B.J., Mayo M.S.: Bayesian sample size calculations in phase II clinical trials using a mixture of informative priors. Stat. Med. 25, 2554–2566 (2006)
Gajewski B.J., Sedwick J.D., Antonelli P.J.: A log-normal distribution model of the effect of bacteria and ear fenestration on hearing loss: a Bayesian approach. Stat. Med. 23, 493–508 (2004)
Gajewski B.J., Hart S., Bergquist S., Dunton N.: Inter-rater reliability of pressure ulcer staging: ordinal probit Bayesian hierarchical model that allows for uncertain rater response. Stat. Med. 26(25), 4602–4618 (2007)
Gajewski B.J., Nicholson N., Widen J.E.: Predicting hearing threshold in non-responsive subjects using a log-normal Bayesian linear model in the presence of left censored covariates. Stat. Biopharm. Res. 1(2), 137–148 (2009)
Gajewski B.J., Boyle D.K., Thompson S.: How a Bayesian might estimate the distribution of Cronbach’s alpha from ordinal-dynamic scaled data: a case study measuring nursing home residents quality of life. Methodol. Eur. J. Res. Methods Behav. Soc. Sci. 2, 71–82 (2010)
Garthwaite P.H., Kadane J.B., O’Hagan A.: Statistical methods for eliciting probability distributions. J. Am. Stat. Assoc. 100, 680–701 (2005)
Gelman A., Carlin J.B., Stern H.S., Rubin D.B.: Bayesian Data Analysis. Chapman and Hall, New York (2004)
Genest C., McConway K.J.: Allocating weights in the linear opinion pool. J. Forecast. 9, 53–73 (1990)
Gilks W.R., Thomas A., Spiegelhalter D.J.: A language and program for complex Bayesian modeling. Statistician 43, 169–177 (1994)
Goodwin L.D.: Changing conceptions of measurement validity: an update on the new standards. J. Nurs. Educ. 41(3), 100–106 (2002)
Gosling J.P., Oakley J.E., O’Hagan A.: Nonparametric elicitation for heavy-tailed prior distributions. Bayesian Anal. 2(4), 693–718 (2007)
Grant J.S., Davis L.L.: Selection and use of content experts for instrument development. Res. Nurs. Health 20, 269–274 (1997)
Haynes S.N., Richard D.C., Kubany E.S.: Content validity in psychological assessment: functional approach to concepts and methods. Psychol. Assess. 7(3), 238–247 (1995)
Johnson V.E., Albert J.H.: Ordinal data modeling. Springer-Verlag, New York (1999)
Kane, M.T.: Validity. In: Brennan, R.L. (ed.) Educational Measurement, pp. 30–38. American Council on Education/Praeger Series on Higher Education, New York (2006)
Lee S.Y.: A Bayesian approach to confirmatory factor analysis. Psychometrika 46, 153–160 (1981)
Lee S.Y.: Structural equation modeling: A Bayesian approach. Wiley, New York (2007)
Lee S.Y., Song X.Y.: Evaluation of the Bayesian maximum likelihood approaches to analyzing structural equation models with small sample sizes. Multivar. Behav. Res. 39, 653–686 (2004)
Little T.D., Lindenberger U., Nesselroade J.R.: On selecting indicators for multivariate measurement with latent variables: When ‘good’ indicators are bad and when ‘bad’ indicators are good. Psychol. Methods 4, 192–211 (1999)
Messick, S.: Validity. In: Linn, R.L. (ed.) Educational Measurement, pp. 13–103. American Council on Education and Macmillian, New York (1989)
Muthén, B.: Bayesian analysis in Mplus: A brief introduction. Technical Report, http://www.statmodel.com/download/IntroBayesVersion%203.pdf(2010)
Nunnally, J.C., Bernstein, I.H.: Pyschometric Theory. 3rd edn. McGraw-Hill, New York (1994)
O’Hagan A.: Eliciting expert beliefs in substantial practical applications. Statistician 47, 21–35 (1998)
O’Hagan A., Buck C.E., Daneshkhah A., Eiser J.R., Garthwaite P.H., Jenkinson D.J., Oakley J.E., Rakow T.: Uncertain Judgments: eliciting experts’ probabilities. Wiley, New York (2006)
Pett M.A., Lackley N.R., Sullivan J.J.: Making sense of factor analysis: the Use of Factor Analysis for Instrument Development in Health Care Research. Sage, Thousand Oaks (2003)
Polit D.F., Beck C.T.: The content validity index: are you sure you know what’s being reported? Critique and recommendations. Res. Nurs. Health 29, 489–497 (2006)
Reese C.S., Wilson A.G., Hamada M.S., Martz H.F., Ryan K.J.: Integrated analysis of computer and physical experiments. Technometrics 46, 153–164 (2004)
Rejeski W.J., Ip E.H., Marsh A.P., Miller M.E., Farmer D.F.: Measuring disability in older adults: the International classification system of functioning, disability and health (ICF) framework. Geriatr. Gerontol. Int. 8, 48–54 (2008)
Rubio D.M., Berg-Weger M., Tebb S.S., Lee E.S., Rauch S.: Objectifying content validity: conducting a content validity study in social work research. Soc. Work Res. 27(2), 94–104 (2003)
Samaniego F.J., Reneau D.M.: Toward a reconciliation of the Bayesian and Frequentist approaches to point estimation. J. Am. Stat. Assoc. 89(427), 947–957 (1994)
Sinharay S., Johnson M.S., Stern H.S.: Posterior predictive assessment of item response theory models. Appl. Psychol. Meas. 30, 298–321 (2006)
Sireci S.C., Geisinger K.F.: Analyzing test content using cluster analysis and multidimensional scaling. Appl. Psychol. Meas. 16(1), 17–31 (1992)
Sireci S.C., Geisinger K.F.: Using subject-matter experts to assess content representation: an MDS analysis. Appl. Psychol. Meas. 19(3), 17–31 (1995)
Spiegelhalter D.J., Abrams K.R., Myles J.P.: Bayesian Approaches to Clinical Trials and Health-Care Evaluation. Wiley, New York (2004)
Thompson S., Bott M., Boyle D., Gajewski B., Tilden V.P.: A measure of palliative care in nursing homes. J. Pain Symptom Manag. 41(1), 57–67 (2011)
Wilks S.S.: Mathematical Statistics. Wiley, New York (1962)
Wirth R.J., Edwards M.C.: Item factor analysis: current approaches and future directions. Psychol. Methods 12, 58–79 (2007)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Gajewski, B.J., Price, L.R., Coffland, V. et al. Integrated analysis of content and construct validity of psychometric instruments. Qual Quant 47, 57–78 (2013). https://doi.org/10.1007/s11135-011-9503-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11135-011-9503-4