Abstract
In the past decade, there has been a proliferation of community- and school-based college readiness programs designed to increase the participation of students who have traditionally been underrepresented in higher education. However, few of these college readiness programs have been empirically evaluated. This study examines the impact of one such intervention, the College Bound, St. Louis (CB) program. Using propensity weighting and doubly robust modeling, we found CB participants were more likely to reach proficiency on the End of Course exams, to obtain at least a B grade in a number of foundational college courses, to take more AP or honors courses, and to attend a 4-year postsecondary institution than similarly situated non-participants. Future directions for evaluating similar college readiness programs are discussed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Minorities, low-income, and first-generation college students are not mutually exclusive groups. Studies have shown positive intercorrelations among membership within these groups, such that first-generation college students are more likely to be African-Americans, Hispanic, and come from low-income families (Darling and Smith 2007; Engle and Tinto 2008). Similarly, Hispanics and African-Americans are overrepresented among the population of low-income college students (Cho et al. 2013).
The median p values and proportion of rejections for individual categories within a baseline covariate may not exactly coincide. For each outcome, only the observations for which the outcome is observed are retained. Thus, some individual baseline variable categories may not be observed for all outcomes, allowing for differing values across categories within a covariate in the summary table displayed.
References
Achieve. (2004). Ready or not: Creating a high school diploma that counts. Washington, DC: Achieve, Inc.
ACT. (2012). 2012 ACT national and state scores. National score trends. Iowa City, IA: ACT.
ACT. (2013). Readiness matters: The impact of college readiness on college persistence and degree completion. Iowa City, IA: ACT.
Adelman, C. (2006). The toolbox revisited: Paths to a degree completion from high school through college. Washington, D.C.: U.S. Department of Education, Office of Research and Improvement.
Allensworth, E., Nomi, T., Montgomery, N., & Lee, V. F. (2009). College preparatory curriculum for all: Academic consequences of requiring Algebra and English I for ninth graders in Chicago. Educational Evaluation and Policy Analysis, 31(4), 367–391. doi:10.3102/0162373709343471.
American Community Survey. (2013). American Community Survey 2013. Washington, DC: U.S. Census Bureau. http://factfinder.census.gov/faces/nav/jsf/pages/index.xhtml.
Avery, C., & Kane, T. J. (2004). Student perceptions of college opportunities: The Boston Coach Program. In C. M. Hoxby (Ed.), College choices: The economics of where to go, when to go, and how to pay for it (pp. 355–394). Chicago, IL: University of Chicago Press.
Bailey, T., Jeong, D. W., & Cho, S. (2010). Referral, enrollment, and completion in developmental education sequences in community colleges. Economics of Education Review, 29, 255–270. doi:10.1016/j.econedurev.2009.09.002.
Balemian, K. & Feng, J. (2013). First generation students: College aspirations, preparedness and challenges. In Paper presented at the College Board AP Conference, Las Vegas, NV.
Bang, H., & Robins, J. M. (2005). Doubly robust estimation and missing data and causal inference models. Biometrics, 61(4), 962–972. doi:10.1111/j.1541-0420.2005.00377.xBarnett.
Barnett, E. A., Corrin, W., Nakanishi, A., Bork, R. H., Mitchell, C., & Sepanik, S. (2012). Preparing high school students for college: An exploratory study of college readiness partnership programs in Texas. Washington, DC: National Center for Postsecondary Research.
Baum, S., Ma, J., & Payea, K. (2013). Education pays 2013: The benefits of education for individuals and society. New York, NY: College Board.
Belasco, A. S. (2013). Creating college opportunity: School counselors and their influence on postsecondary enrollment. Research in Higher Education, 54(7), 781–804. doi:10.1007/s11162-013-9297-4.
Belfield, C. R., & Crosta, P. M. (2012). Predicting success in college: The importance of placement tests and high school transcripts. New York, NY: Community College Research Center Teachers College, Columbia University.
Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 57(1), 289–300. doi:10.2307/2346101.
Bowen, W. G., Chingos, M. M., & McPherson, M. S. (2009). Crossing the finish line: Completing college at America’s public universities. Princeton, NJ: Princeton University Press.
Buchmann, C., Condron, D. J., & Roscigno, V. J. (2010). Shadow education, American style: Test preparation, the SAT, and college enrollment. Social Forces, 89(2), 435–461. doi:10.1353/sof.2010.0105.
Cabrera, N. L., Miner, D. D., & Milem, J. F. (2013). Can a summer bridge program impact first-year persistence and performance? A case study of the New Start Summer Program. Research in Higher Education, 54, 481–498. doi:10.1007/s11162-013-9286-7.
Carnevale, A. P., Jayasundera, T., & Chea, B. (2012). The college advantage: Weathering the economic storm. Washington, DC: The Georgetown University Center on Education and the Workforce.
Carnevale, A. P., Rose, S. J., & Cheah, B. (2011). The college payoff: Education, occupations, lifetime earnings. Washington, DC: The Georgetown University Center on Education and the Workforce.
Carrell, S. E., & Sacerdote, B. (2013). Late interventions matter too: The case of College Coaching New Hampshire. Cambridge, MA: National Bureau of Economic Research.
Cho, S. W., Jacobs, J., & Zhang, C. (2013). Demographic and academic characteristics of Pell grant recipients at community colleges. New York, NY: Community College Research Center at Teachers College, Columbia University.
Complete College America. (2011). Time is the enemy. Indianapolis, IN: Complete College America.
Conger, D., Long, M. C., & Iatarola, P. (2009). Explaining race, gender, and poverty disparities in advanced course-taking. Journal of Policy Analysis and Management, 28(4), 555–576. doi:10.1002/pam.20455.
Conley, D. T. (2008). What makes a student college ready. Educational Leadership, 66(2), 1–3.
Darling, R. A., & Smith, M. S. (2007). First-generation college students: First-year challenges. In M. S. Hunter, B. McCalla-Wriggins, & E. R. White (Eds.), Academic advising: New insights for teaching and learning in the first year (pp. 203–211). Manhattan, KS: NACADA Monographs.
DeGiorgi, G., Pellizzari, M., & Redaelli, S. (2010). Identification of social interactions through partially overlapping peer groups. American Economic Journal: Applied Economics, 2(2), 241–275. doi:10.1257/app.2.2.241.
Domina, T. (2009). What works in college outreach: Assessing targeted and schoolwide interventions for disadvantaged students. Educational Evaluation and Policy Analysis, 31(2), 127–152. doi:10.3102/0162373709333887.
Engle, J. (2007). Postsecondary access and success for first-generation college students. In P. J. Beauvais (Ed.), American academic (Vol. 3, pp. 25–48). Washington, DC: American Federation of Teachers.
Engle, J., & Tinto, V. (2008). Moving beyond access: College success for low-income, first-generation students. Washington, DC: Pell Institute.
Feeney, M. & Heroff, J. (2013). Barriers to need-based financial aid: Predictors of timely FAFSA completion among low-income students. Journal of Student Financial Aid, 43(2), 65–85. http://publications.nasfaa.org/jsfa/vol43/iss2/2.
Feinstein, O. (2012). Evaluation as a learning tool. New Directions for Evaluation, 2012(134), 103–112. doi:10.1002/ev.20022.
Fischer, M. J. (2007). Settling into campus life: Differences by race/ethnicity in college involvement and outcomes. The Journal of Higher Education, 78(2), 125–161. doi:10.1353/jhe.2007.0009.
Gandara, P., & Bial, D. (2001). Paving the way to postsecondary education: K-12 intervention programs for underrepresented youth. Washington, DC: National Center for Education Statistics, U.S. Department of Education.
Gibbons, M. M., & Borders, L. D. (2010). Prospective first-generation college students: A social-cognitive perspective. Career Development Quarterly, 58(3), 194–208. doi:10.1002/j.2161-0045.2010.tb00186.x.
Grodsky, E., & Jones, M. T. (2007). The real and imagined barriers to college entry: Perceptions of college costs. Social Science Research, 36(2), 745–766. doi:10.1016/j.ssresearch.2006.05.001.
Gullatt, Y., & Jan, W. (2003). How do pre-collegiate academic outreach programs impact college-going among underrepresented students?. Washington, DC: Pathways to College Network.
Harder, V. S., Stuart, E. A., & Anthony, J. (2010). Propensity score techniques and the assessment of measured covariate balance to test causal association in psychological research. Psychological Methods, 15(3), 234–249. doi:10.1037/a0019623.
Hein, V., Smerdon, B., & Sambolt, M. (2013). Predictors of postsecondary success. Washington, DC: American Institutes for Research.
Hill, L. D., Bregman, A., & Andrade, F. (2015). Social capital for college: Network composition and access to selective institutions among urban high school students. Urban Education, 50(3), 316–345. doi:10.1177/0042085913514590.
Horn, L., Chen, X., & Chapman, C. (2003). Getting ready to pay for college: What students and their parents know about the cost of college tuition and what they are doing to find out. Washington, DC: National Center for Education Statistics.
Hoxby, C., & Avery, C. (2013). The missing one-offs: The hidden supply of high-achieving low-income students. In Brookings papers on economic activity (pp. 1–65).
Institute for Higher Education Policy. (2010). A portrait of low-income young adults in education. Washington, DC: Institute for Higher Education Policy.
Kena, G., Aud, S., Johnson, F., Wang, X., Zhang, J., Rathbun, A., et al. (2014). The condition of education 2014 (NCES 2014-083). Washington, DC: U.S. Department of Education, National Center for Education Statistic.
Klasik, D. (2012). The college application gauntlet: A systematic analysis of the steps to four-year college enrollment. Research in Higher Education, 53(5), 506–549. doi:10.1007/s11162-011-9242-3.
Lasilla, N. E. (2011). Effects of tuition price, grant aid, and institutional revenue on low-income student enrollment. Journal of Student Financial Aid, 41(3), 24–41.
Lee, V. E. (2002). Restructuring high schools for equity and excellence: What works. New York: Teachers College Press.
Lee, B. K., Lessler, J., & Stuart, E. (2010). Improving propensity score weighting using machine learning. Statistics in Medicine, 29(3), 337–346. doi:10.1002/sim.3782.
Lehmann, W. (2007). I just didn’t feel like I fit in: The role of habitus in university dropout decisions. Canadian Journal of Higher Education, 37(2), 89–110.
Long, M. C., Conger, D., & Iatarola, P. (2012). Effects of high school course-taking on secondary and postsecondary success. American Educational Research Journal, 49(2), 285–322. doi:10.3102/0002831211431952.
Long, B. T., & Riley, E. (2007). Financial aid: A broken bridge to college access? Harvard Educational Review, 77(1), 39–63.
Luna De La Rosa, M. L. (2006). Is opportunity knocking? Low-income students’ perceptions of college and financial aid. American Behavioral Scientist, 49(12), 1670–1686. doi:10.1177/0002764206289139.
McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004). Propensity score estimate with boosted regression for evaluating causal effects in observational studies. Psychological Methods, 9(4), 403–425. doi:10.1037/1082-989X.9.4.403.
Mitra, D. (2011). Pennsylvania’s best investment: The social and economic benefits of public education. State College, PA: Pennsylvania State University.
Moschetti, R., & Hudley, C. (2008). Measuring social capital among first-generation and non-first-generation, working-class, white males. Journal of College Admission, 198, 25–30.
Nagaoka, J., Roderick, M., & Coca, V. (2009). Barriers to college attainment: Lessons from Chicago. Center for American Progress DC. Washington, DC: Associated Press.
National Center for Health Statistics. (2012). Health, United States, 2011. With special features on socioeconomic status and health. Hyattsville, MD: National Center for Health Statistics.
National Commission on Excellence in Education. (1983). A nation at risk: The imperative for educational reform. Washington, DC: U.S. Department of Education.
New England Board of Higher Education. (2012). Policy snapshot: Assessing and increasing college readiness in New England. Boston, MA: New England Board of Higher Education.
Nunez, A. M. (2009). Modeling the effects of diversity experiences and multiple capitals on Latina/o college students’ academic self-confidence. Journal of Hispanic Higher Education, 8(2), 179–196. doi:10.1177/1538192708326391.
Olshansky, S. J., Antonucci, T., Berkman, L., Binstock, R. H., Boersch-Supan, A., Cacioppo, J. T., et al. (2012). Differences in life expectancy due to race and educational differences are widening, and many may not catch up. Health Affairs, 31(8), 1803–1813. doi:10.1377/hlthaff.2011.0746.
Pascarella, E. T., Pierson, C. T., Wolniak, G. C., & Terenzini, P. T. (2004). First generation college students: Additional evidence on college experiences and outcomes. The Journal of Higher Education, 75(3), 249–284. doi:10.1353/jhe.2004.0016.
Pathways to College Network. (2004). A shared agenda: A leadership challenge to improve college access and success. Boston: Pathways to College Network.
Pell Institute. (2011). Pell Institute fact sheet. Washington, DC: Pell Institute.
Pell Institute and the University of Pennsylvania Alliance for Higher Education and Democracy. (2015). Indicators of higher education equity in the United States: 45 year trend report. Washington, DC: Pell Institute.
Perna, L. W., & Swail, S. (2001). Pre-college outreach and early intervention. Thought & Action, 17(1), 99–110.
Perna, L. W., & Titus, M. A. (2005). The relationship between parental involvement as social capital and college enrollment: An examination of racial/ethnic group differences. The Journal of Higher Education, 76(5), 485–518. doi:10.1353/jhe.2005.0036.
Pew Research Center. (2010). The reversal of the college marriage gap. Washington, DC: Pew Research Center.
Pew Research Center. (2014). The rising cost of not going to college. Washington, DC: Pew Research Center.
Pike, G. R., & Kuh, G. D. (2005). First- and second-generation college students: A comparison of their engagement and intellectual development. The Journal of Higher Education, 76(3), 276–300. doi:10.1353/jhe.2005.0021.
Ridgeway, G., Madigan, D., & Richardson, T. (1999). Boosting methodology for regression problems. In D. Heckerman & J. Whittaker (Eds.), Proceedings of Artificial Intelligence and Statistics’99 (pp. 152–161). San Francisco, CA: Morgan Kaufmann.
Roderick, M., Coca, V., & Nagaoka, J. (2011). Potholes on the road to college: High school effects in shaping urban students’ participation in college application, four-year college enrollment, and college match. Sociology of Education, 84(3), 178–211. doi:10.1177/0038040711411280.
Roderick, M., Nagaoka, J., & Coca, V. (2009). College readiness for all: The challenge for urban high schools. The Future of Children, 19(1), 185–210. doi:10.1353/foc.0.0024.
Ross, T., Kena, G., Rathbun, A., KewalRamani, A., Zhang, J., Kristapovich, P., & Manning, E. (2012). Higher education: Gaps in access and persistence study. Washington, DC: U.S. Department of Education, National Center of Education Statistics.
Ryu, M. (2015). Basic facts about US higher education today. Washington, DC: American Council on Education and Center for Policy Research and Strategy.
Saenz, V. B., Hurtado, S., Barrera, D., Wolf, D., & Yeung, F. (2007). First in my family: A profile of first-generation college students at four-year institutions since 1971. Los Angeles, CA: Higher Education Research Institute at UCLA.
Schafer, J. L. (1997). Analysis of incomplete multivariate data. London: Chapman & Hall.
Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177. doi:10.1037/1082-989X.7.2.147.
Seftor, N. S., Mamun, A., & Schirm, A. (2009). The impacts of regular upward bound on postsecondary outcomes seven to nine years after scheduled high school graduation. Princeton, NJ: Mathematica Policy Research.
Shaw, E. J., Marini, J. P., & Mattern, K. D. (2013). Exploring the utility of Advanced Placement participation and performance in college admission decisions. Educational and Psychological Measurement, 73(2), 229–253. doi:10.1177/0013164412454291.
Smith, J., Pender, M., & Howell, J. (2013). The full extent of student-college undermatch. Economics of Education Review, 32, 247–261. doi:10.1016/j.econedurev.2012.11.001.
Sparks, D., & Malkus, N. (2013). First-year undergraduate remedial coursetaking: 1999–2000, 2003–2004, 2007–2008. Washington, DC: National Center for Education Statistics, U.S. Department of Education.
Standing, K., Judkins, D., Keller, B., & Shimshak, A. (2008). Early outcomes of the GEAR UP program: Final report. Washington, DC: U.S. Department of Education, Office of Planning, Evaluation and Policy Development.
Stanton-Salazar, R. D. (2010). A social capital framework for the study of institutional agents and their role in the empowerment of low-status students and youth. Youth & Society, 43(3), 1066–1110. doi:10.1177/0044118X10382877.
Swail, W. S., & Perna, L. W. (2002). Pre-college outreach programs: A National Perspective. In W. G. Tierney & L. S. Hagedorn (Eds.), Increasing access to college: Extending possibilities to all students (pp. 15–34). Albany, NY: State University of New York.
Swail, W. S., Quinn, K., Landis, K., & Fung, M. (2012). 2012 National survey of pre-college outreach programs. Virginia Beach, VA: Education Policy Institute.
Trust, Education. (2013). Finding America’s missing AP and IB students. Washington, DC: Education Trust.
Turner, S. E. (2004). Going to college and finishing college: Explaining different educational outcomes. In C. M. Hoxby (Ed.), College choices: The economics of where to go, when to go, and how to pay for it (pp. 13–61). Cambridge, MA: National Bureau of Economic Research.
Zhang, Q., & Sanchez, E. I. (2013). High school grade inflation from 2004 to 2011. Iowa City, IA: ACT.
Acknowledgments
We would like to thank TG for funding this study. We are also grateful to Lisa Orden Zarin, Meesa Olah, Nicole Rainey, Laurie Bainter, and all of the College Bound staff and coaches for their assistance and feedback, which considerably improved this article. However, any errors remain our own.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
Depending on the outcome, the analysis included between 71 and 2031 comparison students, with a median of 951 comparison students per analysis. For CB students, the analysis included between 58 and 321 students, with a median of 179 CB participants per analysis. Tables 6 and 7 provide a summary of the covariate balance between CB participants and the comparison students for the continuous and categorical variables, respectively. For both tables, the first column provides the covariate, and the next three columns provide the mean or percent for the comparison group prior to weighting, the mean or percent for the comparison group after weighting, and the mean or percent for the CB participants, respectively. For the continuous variables, the table also provides the median K–S statistic, with lower values indicating greater concurrence of the CB and weighted comparison distributions, and the proportion of times that the chi square test associated with the K–S statistic was rejected, as well as the median p value for the test that the CB and weighted comparison means are the same and the proportion of times that test was rejected. For the categorical variables, the table indicates the median difference between the proportion of CB and weighted comparison observations found in each category, the median p value for a weighted chi square test of independence between the baseline covariate and CB participation and the proportion of times this test was rejected.Footnote 2
It is important to recognize that for each outcome, there are multiple values associated with the displayed balance measure stemming from the imputation process, where we created 10 sets of plausible values. For each of these 10 separate imputations, there were 10 separate balance tables for each outcome. To synthesize the information across these covariate balance tables within an outcome, we found the median for the K–S statistics, differences in proportions, and p values across these 10 imputations. To synthesize the covariate balance information across the multiple outcomes for display in the tables below, we then found the median of these values across all the outcome measures. Due to the multiple imputation process, it is possible that a covariate was balanced in one set of imputation, but not another set of imputation. Similarly, a covariate could be balanced for one outcome but not another outcome. To take into account the variability in covariate balance, we computed the proportion of times the p value was rejected across imputations and across outcomes. If the covariates were balanced across all outcomes and all sets of imputations, the proportion of times the p value was rejected would be zero. Thus, the more often the p values are rejected, the poorer our covariate balance.
As shown in Tables 6 and 7, our covariate balance was mixed, with some variables showing optimal balance (e.g., grade 8 attendance), whereas other variables were less balanced (e.g., disability status). We observed poorer covariate balance on the categorical variables than on the continuous variables, in part because some of the distributions for the categorical variables were skewed. Furthermore, in many instances where the p values were frequently rejected (thereby indicating poor covariate balance), the differences between the groups were small. For example, the results suggested that we could not achieve covariate balance between CB students and the comparison group on the graduating cohort variables, particularly with respect to the 2013–2014 cohort, where the p value was rejected approximately 85 % of the time. Yet, there was only a one percentage point difference between the post-weighted comparison group and the CB participants (i.e., 10.5 % of the post-weighted comparison group were from the 2013–2014 cohort compared to 11.7 % of the CB students). Thus, even though some covariates were not statistically balanced, the differences were not necessarily large either.
Rights and permissions
About this article
Cite this article
Le, VN., Mariano, L.T. & Faxon-Mills, S. Can College Outreach Programs Improve College Readiness? The Case of the College Bound, St. Louis Program. Res High Educ 57, 261–287 (2016). https://doi.org/10.1007/s11162-015-9385-8
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11162-015-9385-8