Abstract
A relatively simple way of measuring institutional effectiveness in relation to degree completion is to estimate the difference between an actual and predicted graduation rate, but the reliability and validity of this method have not been thoroughly examined. Longitudinal data were obtained from IPEDS for both public and private not-for-profit 4-year institutions (n = 1496). Hierarchical panel regression was used to predict 4- and 6-year graduation rates based on structural, demographic, financial, and contextual attributes. A direct effects model yielded effectiveness scores that were highly correlated between consecutive data years (r = 0.65–0.80), which indicated acceptable to good test–retest reliability. A test of convergent validity indicated that effectiveness scores were positively associated with students’ perceptions of a supportive campus environment (r = 0.32–0.45). A test of discriminant validity revealed relatively small correlations between effectiveness scores and institutional attributes, such as educational expenditures (r = 0.07–0.16). The modeling of interaction effects in relation to institutional type marginally improved the validity of effectiveness scores among public but not private institutions. The results suggest that correct model specification can yield residual scores that reliably and validly measure institutional effectiveness in promoting timely degree completion.
Similar content being viewed by others
Notes
The New Economy Index reflects economic performance in the domains of knowledge jobs, globalization, economic dynamism, the digital economy, and innovation capacity (Atkinson and Andes 2010).
While both the ACT and SAT are used to assess academic preparedness, differences in their content permit only a rough approximation of concordance (ACT 2012).
The NSSE custom analysis service does not permit researchers to examine potential sources of correlation attenuation. In order to examine this possibility, data were collected from 53 public institutions that had publicly accessible data from the 2004 NSSE administration. After removing two high-leverage outliers, the correlation between the NSSE SCE scale and the institutional effectiveness scores was very strong, r = 0.73.
References
ACT. (2012). ACT-SAT concordance. Retrieved from http://www.act.org/aap/concordance/.
Adelman, C. (2006). The toolbox revisited: Paths to degree completion from high school through college. Washington, DC: U.S. Department of Education.
American Association of State Colleges and Universities. (2002). Accountability and graduation rates: Seeing the forest and the trees. Retrieved from http://archive.sheeo.org/access/Graduation%20Rate%20Brief%20-%20AASCU.pdf.
Astin, A. W. (1993). What matters in college: Four critical years revisited. San Francisco: Jossey-Bass.
Astin, A. W. (2006). Making sense out of degree completion rates. Journal of College Student Retention: Research, Theory & Practice, 7(1–2), 5–17.
Astin, A. W., & Oseguera, L. (2005). Degree attainment rates at American colleges and universities. Los Angeles: Higher Education Research Institute, UCLA.
Aten, B. H., Figueroa, E. B., & Martin, T. M. (2012). Regional price parities for states and metropolitan areas, 2006-2010. Retrieved from http://www.bea.gov/scb/pdf/2012/08%20August/0812_regional_price_parities.pdf.
Atkinson, R. D., & Andes, S. (2010). The 2010 state new economy index. Retrieved from http://www.itif.org/files/2010-state-new-economy-index.pdf.
Bailey, T., Calcagno, J. C., Jenkins, D., Leinbach, T., & Kienzi, G. (2006). Is student-right-to-know all you should know? An analysis of community college graduation rates. Research in Higher Education, 47(5), 491–519.
Bailey, T., & Xu, D. (2012). Input-adjusted graduation rates and college accountability: What is known from twenty years of research? Retrieved from http://www.hcmstrategists.com/contextforsuccess/papers/LIT_REVIEW.pdf.
Berger, J. B., & Milem, J. F. (2000). Organizational behavior in higher education and student outcomes. In J. C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. XV, pp. 268–338). New York: Springer.
Campbell, C. M., & Cabrera, A. F. (2011). How sound is NSSE? Investigating the psychometric properties of NSSE at a public, research-extensive institution. The Review of Higher Education, 35(1), 77–103.
Chen, R. (2012). Institutional characteristics and college student dropout risks: A multilevel event history analysis. Research in Higher Education, 53(5), 487–505.
Choy, S. (2002). Nontraditional undergraduates: Findings from “The Condition of Education, 2002.” Retrieved from http://files.eric.ed.gov/fulltext/ED471077.pdf.
Corman, H., & Davidson, P. K. (1984). Economic aspects of post-secondary schooling decisions. Economics of Education Review, 3(2), 131–139.
Dougherty, K. J., Natow, R. S., & Vega, B. E. (2012). Popular but unstable: Explaining why state performance funding systems in the United States often do not persist. Teachers College Record, 114, 1–41.
Field, K. (2013, August 22). Obama plan to tie student aid to college ratings draws mixed reviews. Chronicle of Higher Education. Retrieved from http://chronicle.com/article/Obama-Plan-to-Tie-Student-Aid/141229/.
Furr, M. (2011). Scale construction and psychometrics for social and personality psychology. Thousand Oaks, CA: SAGE Publications.
Gansemer-Topf, A. M., & Schuh, J. H. (2006). Institutional selectivity and institutional expenditures: Examining organizational factors that contribute to retention and graduation. Research in Higher Education, 47(6), 613–642.
Goenner, C. F., & Snaith, S. M. (2004). Accounting for model uncertainty in the prediction of university graduation rates. Research in Higher Education, 45(1), 25–41.
Gordon, J., Ludlum, J., & Hoey, J. J. (2008). Validating NSSE against student outcomes: Are they related? Research in Higher Education, 49(1), 19–39.
Hemphill, J. F. (2003). Interpreting the magnitudes of correlation coefficients. American Psychologist, 58(1), 78–80.
Horn, L. (1999). Stopouts or stayouts? Undergraduates who leave college in their first year. Retrieved from http://nces.ed.gov/pubs99/1999087.pdf.
Horn, A. S. (2013). The selection of peer states for performance benchmarking in higher education. Retrieved from http://www.mhec.org/sites/mhec.org/files/20130429peer_state_selection_working_paper.pdf.
Horn, L., & Carroll, C. D. (2006). Placing college graduation rates in context: How 4-year college graduation rates vary with selectivity and the size of low-income enrollment. Retrieved from http://nces.ed.gov/pubs2007/2007161.pdf.
Humphreys, D. (2012). What’s wrong with the completion agenda—And what we can do about it. Retrieved from https://www.aacu.org/liberaleducation/le-wi12/humphreys.cfm.
Kamens, D. H. (1971). The college” charter” and college size: Effects on occupational choice and college attrition. Sociology of Education, 44, 270–296.
Kelchen, R., & Harris, D. (2012). Can value-added methods improve the measurement of college performance? Empirical analyses and policy implications. Retrieved from http://www.hcmstrategists.com/contextforsuccess/papers/KELCHEN_HARRIS_PAPER.pdf.
Kuh, G. D., Cruce, T. M., Shoup, R., & Kinzie, J. (2008). Unmasking the effects of student engagement on first-year college grades and persistence. The Journal of Higher Education, 79(5), 540–563.
McCormick, A. C., Pike, G. R., Kuh, G. D., & Chen, P. D. (2009). Comparing the utility of the 2000 and 2005 Carnegie classification systems in research on students’ college experiences and outcomes. Research in Higher Education, 50(1), 144–167.
McGuinness, A. C. (2011). The states and higher education. In P. G. Altbach, P. J. Gumport, & R. O. Berdahl (Eds.), American higher education in the twenty-first century (pp. 139–169). Baltimore: Johns Hopkins University Press.
Michell, J. (2003). Epistemology of measurement: The relevance of its history for quantification in the social sciences. Social Science Information, 42, 515–534.
Monks, J. (2000). The returns to individual and college characteristics: Evidence from the National Longitudinal Survey of Youth. Economics of Education Review, 19(3), 279–289.
Muthén, B., & Satorra, A. (1995). Complex sample data in structural equation modeling. In P. Marsden (Ed.), Sociological methodology (pp. 216–316). Cambridge: Blackwell.
National Survey of Student Engagement. (2007). Reliability: 2007 Institution-level temporal stability. Retrieved from http://nsse.iub.edu/pdf/psychometric_portfolio/Reliability_TemporalStability_Institution-Level_2007.pdf.
National Survey of Student Engagement. (2012). Promoting student learning and institutional improvement: Lessons from NSSE at 13. Bloomington, IN: Indiana University Center for Postsecondary Research.
NCHEMS. (2013). Workforce conditions. Retrieved from http://www.higheredinfo.org/.
Nunnally, J. C. (1988). Psychometric theory. Englewood Cliffs, NJ: McGraw-Hill.
Oseguera, L. (2005). Four and six-year baccalaureate degree completion by institutional characteristics and racial/ethnic groups. Journal of College Student Retention, 7(1–2), 19–59.
Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students. San Francisco: Jossey-Bass.
Pike, G. R. (2013). NSSE benchmarks and institutional outcomes: A note on the importance of considering the intended uses of a measure in validity studies. Research in Higher Education, 54(2), 149–170.
Pike, G. R., Kuh, G. D., McCormick, A. C., Ethington, C. A., & Smart, J. C. (2011). If and when money matters: The relationships among educational expenditures, student engagement and students’ learning outcomes. Research in Higher Education, 52(1), 81–106.
Pike, G. R., Smart, J. C., Kuh, G. D., & Hayek, J. C. (2006). Expenditures and student engagement: When does money matter? Research in Higher Education, 47(7), 847–872.
Porter, S. R. (2000). The robustness of the graduation rate performance indicator used in the U.S. News & World Report college rankings. The Case International Journal of Educational Advancement, 1(2), 145–164.
Porter, S. R. (2011). Do college student surveys have any validity? The Review of Higher Education, 35(1), 45–76.
Price, D. V., & Tovar, E. (2014). Student engagement and institutional graduation rates: Identifying high-impact educational practices for community colleges. Community College Journal of Research and Practice, 38(9), 766–782.
Raghunathan, T. E., Rosenthal, R., & Rubin, D. B. (1996). Comparing correlated but nonoverlapping correlations. Psychological Methods, 1(2), 178.
Rhee, B. (2008). Institutional climate and student departure: A multinomial multilevel modeling approach. The Review of Higher Education, 31(2), 161–183.
Ryan, J. F. (2004). The relationship between institutional expenditures and degree attainment at baccalaureate colleges. Research in Higher Education, 45(2), 97–114.
Scott, M., Bailey, T., & Kienzl, G. (2006). Relative success? Determinants of college graduation rates in public and private colleges in the US. Research in Higher Education, 47(3), 249–279.
Smart, J. C., Ethington, C. A., Riggs, R. O., & Thompson, M. D. (2002). Influences of institutional expenditure patterns on the development of students’ leadership competencies. Research in Higher Education, 43(1), 115–132.
Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87, 245–251.
Stratton, L. S., O’Toole, D. M., & Wetzel, J. N. (2005). A multinomial logit model of college stopout and dropout behavior. Retrieved from http://anon-ftp.iza.org/dp1634.pdf.
The National Center for Public Policy and Higher Education. (2008). Measuring Up 2008: The national report card of higher education. Retrieved from http://measuringup2008.highereducation.org/print/NCPPHEMUNationalRpt.pdf.
Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition. Chicago: The University of Chicago Press.
Titus, M. A. (2004). An examination of the influence of institutional context on student persistence at 4-year colleges and universities: A multilevel approach. Research in Higher Education, 45(7), 673–699.
Toutkoushian, R. K., & Smart, J. C. (2001). Do institutional characteristics affect student gains from college? The Review of Higher Education, 25(1), 39–61.
Volkwein, J. F., & Grunig, S. D. (2005). Resources and reputation in higher education: Double, double, toil and trouble. In J. C. Burke & Associates (Ed.), Achieving accountability in higher education: Balancing public, academic, and market demands (pp. 246–274). San Francisco: Jossey-Bass.
Volkwein, J. F., & Tandberg, D. A. (2008). Measuring up: Examining the connections among state structural characteristics, regulatory practices, and performance. Research in Higher Education, 49(2), 180–197.
von Hippel, P. T. (2007). Regression with missing Y’s: An improved strategy for analyzing multiply imputed data. Sociological Methodology, 37, 83–117.
Webber, D. A., & Ehrenberg, R. G. (2010). Do expenditures other than instructional expenditures affect graduation and persistence rates in American higher education? Economics of Education Review, 29(6), 947–958.
Acknowledgments
The authors would like to acknowledge the helpful comments of Ernest Davenport as well as the able assistance provided by Allison BrckaLorenz.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Horn, A.S., Lee, G. The Reliability and Validity of Using Regression Residuals to Measure Institutional Effectiveness in Promoting Degree Completion. Res High Educ 57, 469–496 (2016). https://doi.org/10.1007/s11162-015-9394-7
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11162-015-9394-7