Abstract
Are the methods of causal inference and, in particular, randomized controlled trials, compatible with the study of political history? While many important questions regarding political institutions and American political development cannot be answered with randomized controlled trials, scholars can and should be using the many instances of randomized experiments conducted by and within government institutions to further our understanding of institutions and political behavior. We argue that a surprising abundance of opportunities are available for scholars to utilize methods of random audits as natural experiments. Public and administrative officials have engaged in randomized interventions or audits to test for policy effects, to encourage compliance with the law, or to distribute government resources or personal risk to citizens fairly. With rare exceptions, such audits have not been leveraged by scholars interested in American political development or political history. Examples of randomized controlled trials conducted by agencies or institutions throughout US history are offered, and a historical random audit of members of the US Congress by the Federal Election Commission is highlighted. We conclude with limitations and advice on how to analyze the effects of randomized controlled trials conducted by governments. Scholars can use historical randomization to enhance causal inference and test theoretical implications, though deep knowledge of descriptive historical data and events are required to discover historical randomizations within political and legal institutions.
Similar content being viewed by others
Notes
Other causal inference methods have been utilized, but we emphasize randomization in our discussion that follows. Examples of other approaches include matching across subjects to estimate the effects of legal changes (e.g., Gordon and Huber 2007; Jensenius 2017; Mattes and Vonnahme 2010); regression discontinuity designs (e.g., Lerman 2014); difference-in-differences with plausibly exogenous institutional changes (O’Brien and Rickne 2016); within-subjects reforms of electoral institutions (e.g., Baskaran and Lopes Da Fonseca 2016; Carson and Sievert 2015; Yoshinaka and Grose 2008); and marginal structural modeling (Torres 2019); and synthetic controls (e.g. Michalski and Wood 2017).
There is evidence that the IRS engages in political targeting (Young, Reksulak, and Shughart 2001), which raises questions as to whether IRS audits are random.
Personal communication with a former FEC Commissioner.
References
Andersen, S. C., & Moynihan, D. P. (2016). Bureaucratic investments in expertise: Evidence from a randomized controlled field trial. Journal of Politics, 78(4), 1032–1044.
Andersen, S. C., & Moynihan, D. P. (2018). How do socially distinctive newcomers fare? Evidence from a field experiment. Public Administration Review, 78(6), 874–882.
Avellaneda, C. N. (2013). Mayoral decision-making: Issue salience, decision context, and choice constraint? An experimental study with 120 Latin American mayors. Journal of Public Administration Research and Theory, 23(3), 631–661.
Ban, P., Fouirnaies, A., Hall, A. B., & Snyder, J. M. (2019). How newspapers reveal political power. Political Science Research and Methods. https://doi.org/10.1017/psrm.2017.43.
Baskaran, T., & Lopes da Fonseca, M. (2016). Electoral thresholds and political representation. Public Choice, 169(1–2), 117–136.
Benston, G. J. (1973). Required disclosure and the stock market: An evaluation of the Securities Exchange Act of 1934. American Economic Review, 63(1), 132–155.
Berinsky, A. J., & Chatfield, S. (2015). An empirical justification for the use of draft lottery numbers as a random treatment in political science research. Political Analysis, 23(3), 449–454.
Broockman, D. E. (2014). Distorted communication, unequal representation: Constituents communicate less to representatives not of their race. American Journal of Political Science, 58(2), 307–321.
Broockman, D. E., & Butler, D. M. (2015). Do better committee assignments meaningfully benefit legislators? Evidence from a randomized experiment in the Arkansas state legislature. Journal of Experimental Political Science, 2(2), 152–163.
Butler, D. M. (2014). Representing the advantaged: How politicians reinforce inequality. New York: Cambridge University Press.
Butler, D. M. (2019). Facilitating field experiments at the subnational level. Journal of Politics, 81(1), 371–376.
Byrnes, P. E., Al-Awadhi, A., Gullvist, B., Brown-Liburd, H., Teeter, R., Warren, J. D., et al. (2012). Evolution of auditing: From the traditional approach to the future audit. In D. Y. Chan, V. Chiu, & M. A. Vasarhelyi (Eds.), Continuous auditing: Theory and application (pp. 285–297). Bingley, UK: Emerald Publishing Limited.
Carson, J. L., & Sievert, J. (2015). Electoral reform and changes in legislative behavior: Adoption of the secret ballot in congressional elections. Legislative Studies Quarterly, 40(1), 83–110.
Chin, M. L., Bond, J. R., & Geva, N. (2000). A foot in the door: An experimental study of PAC and constituency effects on access. Journal of Politics, 62(2), 534–549.
Cirone, A., & Van Coppenolle, B. (2019). Bridging the gap: Lottery-based procedures in early parliamentarization. World Politics, 71(2), 197–235.
Clark, W. R., & Golder, M. (2015). Big data, causal inference, and formal theory: Contradictory trends in political science? PS: Political Science and Politics, 48(1), 65–70.
Costa, M. (2017). How responsive are political elites? A meta-analysis of experiments on public officials. Journal of Experimental Political Science, 4(3), 241–254.
Diamond, J., & Robinson, J. A. (2010). Natural experiments of history. Cambridge, MA: The Belknap Press of Harvard University Press.
Eber, L. (2005). Waiting for Watergate: The long road to FEC reform. Southern California Law Review, 79(5), 1155–1202.
Eldersveld, S. J. (1956). Experimental propaganda techniques and voting behavior. American Political Science Review, 50(1), 154–165.
Erikson, R. S., & Stoker, L. (2011). Caught in the draft: The effects of Vietnam draft lottery status on political attitudes. American Political Science Review, 105(2), 221–237.
Fiorina, M. P., & Plott, C. R. (1978). Committee decisions under majority rule: An experimental study. American Political Science Review, 72(2), 575–598.
Friedman, J. (1996). The rational choice controversy: Economic models of politics reconsidered. New Haven: Yale University Press.
Gerber, A., & Green, D. P. (2000). The effects of canvassing, telephone calls, and direct mail on voter turnout: A field experiment. American Political Science Review, 94(3), 653–663.
Gordon, S. C., & Huber, G. A. (2007). The effect of electoral competitiveness on incumbent behavior. Quarterly Journal of Political Science, 2(2), 107–138.
Gosnell, H. (1927). Getting-out-the-vote: An experiment in the stimulation of voting. Chicago: University of Chicago Press.
Grimmer, J. (2015). We are all social scientists now: How big data, machine learning, and causal inference work together. PS: Political Science and Politics, 48(1), 80–83.
Grose, C. R. (2011). Congress in black and white: Race and representation in Washington and at home. New York: Cambridge University Press.
Grose, C. R. (2014a). Field experimental work on political institutions. Annual Review of Political Science, 17(1), 355–370.
Grose, C. R. (2014b). The adoption of electoral reforms and ideological change in the California state legislature. USC Schwarzenegger Institute Report. http://www.schwarzeneggerinstitute.com/images/SI-Adoption%20of%20Electoral%20Reforms%20Report.pdf. Accessed 5 July 2019.
Grose, C. R. (2019). Experiments, political elites, and political institutions. In J. N. Druckman & D. P. Green (Eds.), Cambridge handbook of experiments. New York: Cambridge University Press.
Grose, C. R., Malhotra, N., & Van Houweling, R. P. (2015). Explaining explanations: How legislators explain their positions and how citizens react. American Journal of Political Science, 59(3), 724–743.
Hall, M. E. K. (2009). Experimental justice: Random judicial assignment and the partisan process of Supreme Court review. American Politics Research, 37(2), 195–226.
Hall, M. E. K. (2010). Randomness reconsidered: Modeling random judicial assignment in the U.S. courts of appeals. Journal of Empirical Legal Studies, 7(3), 574–589.
Hall, A. B., Huff, C., & Kuriwaki, S. (2019). Wealth, slaveownership, and fighting for the confederacy: An empirical study of the American civil war. American Political Science Review, 113(3), 658–673.
Hartman, E., & Hidalgo, F. D. (2018). An equivalence approach to balance and placebo tests. American Journal of Political Science, 62(4), 1000–1013.
Ho, D. E., Handen-Nader, C., Ames, D. & Marcus, D. (2018). Quality review of mass adjudication: A randomized natural experiment at the board of veterans appeals, 2003–2016. Working Paper, Stanford Institute for Economic Policy Research.
Hug, S. (2014). Further twenty years of pathologies? Is rational choice better than it used to be? Swiss Political Science Review, 20(3), 486–497.
Jenkins, J. A. (2012). Studying Congress historically. In J. L. Carson (Ed.), New directions in congressional politics. Routledge/Taylor and Francis Group: New York and London.
Jensenius, F. (2017). Social justice through inclusion: The consequences of electoral quotas in India. New York: Oxford University Press.
Kalla, J. L., & Broockman, D. E. (2016). Campaign contributions facilitate access to congressional officials: a randomized field experiment. American Journal of Political Science, 60(3), 545–558.
Kanthak, K., & Woon, J. (2015). Women don’t run? Election aversion and candidate entry. American Journal of Political Science, 59(3), 595–612.
Karpowitz, C., Preece, J. R., & Monson, J. Q. (2017). How to elect more women: Gender and candidate success in a field experiment. American Journal of Political Science, 61(4), 927–943.
Kastellec, J. P. (2011). Panel composition and voting on the U.S. courts of appeals over time. Political Research Quarterly, 64(2), 377–391.
Kinder, D. R., & Palfrey, T. R. (1993). Experimental foundations of political science. Ann Arbor: University of Michigan Press.
Kriner, D. L., & Schickler, E. (2014). Investigating the president: Committee probes and presidential approval, 1953-2006. Journal of Politics, 76(2), 521–534.
Lajevardi, N. (2018). Access denied: Exploring Muslim American representation and exclusion by state legislators. Politics, Groups, and Identities. https://doi.org/10.1080/21565503.2018.1528161.
Larimer, C. (2018). Voter turnout field experiments. Oxford Bibliographies. https://www.oxfordbibliographies.com/view/document/obo-9780199756223/obo-9780199756223-0243.xml. Accessed 5 July 2019.
Lerman, A. (2014). The modern prison paradox: Politics, punishment, and social community. New York: Cambridge University Press.
Levine, A. S. (2019). Research impact through matchmaking: How and why to connect researchers and practitioners. PS: Political Science and Politics. https://www.r4impact.org/sites/default/files/paper7_r4i_0.pdf. Accessed 5 July 2019.
Levine, D., Toffel, M. W., & Johnson, M. S. (2012). Randomized government safety inspections reduce worker injuries with no detectable job loss. Science, 336(6083), 907–911.
Levy, M. K. (2017). Panel assignment in the federal courts of appeals. Cornell Law Review, 103(1), 65–116.
Mattes, M., & Vonnahme, G. (2010). Contracting for peace: Do nonaggression pacts reduce conflict? Journal of Politics, 72(4), 925–938.
Matthews, D. (2006). A history of auditing: The changing audit process in Britain from the nineteenth century to the present day. New York: Routledge.
Mendez, M. S., & Grose, C. R. (2018). Doubling down: Inequality in responsiveness and the policy preferences of elected officials. Legislative Studies Quarterly, 43(3), 457–491.
Merkel, A., & Vanberg, C. (2019). Legislative bargaining with costly communication. Public Choice. https://doi.org/10.1007/s11127-019-00682-8.
Michalski, R. M., & Wood, A. K. (2017). Twombly and Iqbal at the state level. Journal of Empirical Legal Studies, 14(2), 424–469.
Morton, R. B., & Williams, K. C. (2010). Experimental political science and the study of causality. New York: Cambridge University Press.
O’Brien, D. Z., & Rickne, J. (2016). Gender quotas and women’s political leadership. American Political Science Review, 110(1), 112–126.
Orren, K., & Skowronek, S. (2016). Pathways to the present: Political development in America. In R. Valelly, S. Mettler, & R. Lieberman (Eds.), The Oxford handbook of American political development. New York: Oxford University Press.
Palfrey, T. R. (2008). Laboratory experiments. In D. A. Wittman & B. R. Weingast (Eds.), The Oxford handbook of political economy. New York: Oxford University Press.
Plott, C. R. (2014). Public choice and the development of modern laboratory experimental methods in economics and political science. Constitutional Political Economy, 25(4), 331–353.
Poulos, J. (2019). Land lotteries, long-term wealth, and political selection. Public Choice, 178(1–2), 217–230.
Preece, J. R., & Stoddard, O. B. (2015a). Why women don’t run: Experimental evidence on the role of competition aversion. Journal of Economic Behavior & Organization, 117(1), 296–308.
Preece, J. R., & Stoddard, O. B. (2015b). Does the message matter? A field experiment on political party recruitment. Journal of Experimental Political Science, 2(1), 1–10.
Riker, W. H. (1967). Bargaining in three person games. American Political Science Review, 61(3), 342–356.
Riker, W. H., & Zavoina, W. J. (1970). Rational behavior in politics: Evidence from a three-person game. American Political Science Review, 64(1), 48–60.
Rogowski, J. C., & Sinclair, B. (2017). Estimating the causal effects of social interactions with endogenous networks. Political Analysis, 20(3), 316–328.
Rudolph, S. H. (2005). Perestroika and its other. In K. R. Monroe (Ed.), Perestroika! The raucous rebellion in political science. New Haven: Yale University Press.
Sanders, E. (2005). Work that counts. In K. R. Monroe (Ed.), Perestroika! The raucous rebellion in political science. New Haven, CT: Yale University Press.
Sen, M. (2017). How political signals affect public support for judicial nominations: Evidence from a conjoint experiment. Political Research Quarterly, 70(2), 374–393.
Sunstein, C. R., Schkade, D., Ellman, L. M., & Sawicki, A. (2006). Are judges political? An empirical analysis of the federal judiciary. Washington, DC: Brookings Institution Press.
Titiunik, R. (2016). Drawing your senator from a jar: Term length and legislative behavior. Political Science Research and Methods, 4(2), 293–316.
Titiunik, R., & Feher, A. (2018). Legislative behaviour absent re-election incentives: Findings from a natural experiment in the Arkansas senate. Journal of the Royal Statistical Society, 181(2), 351–378.
Torres, M. (2019). Estimating controlled direct effects through marginal structural models. Political Science Research and Methods. https://www.dropbox.com/s/872jdkg3dgo8ics/MSM_PSRM_2.pdf?dl=0. Accessed 5 July 2019.
Tóth, M., & Chytilek, R. (2018). Fast, frugal, and correct? An experimental study on the influence of time scarcity and quantity of information on the voter decision making process. Public Choice, 177(1–2), 67–86.
Watts, R. L., & Zimmerman, J. L. (1983). Agency problems, auditing, and the theory of the firm: Some evidence. Journal of Law and Economics, 26(3), 613–633.
Weiman, D. F. (1991). Peopling the land by lottery? The market in public lands and the regional differentiation of territory on the Georgia frontier. Journal of Economic History, 51(4), 835–860.
White, A. R., Nathan, N. L., & Faller, J. K. (2015). What do I need to vote? Bureaucratic discretion and discrimination by local election officials. American Political Science Review, 109(1), 129–142.
Wilson, R. K. (2011). The contribution of behavioral economics to political science. Annual Review of Political Science, 14(1), 201–223.
Wood, A. K. (2018). Campaign finance disclosure. Annual Review of Law and Social Science, 14(1), 1–17.
Wood, A. K., Elmendorf, C. S., & Spencer, D. (2019). Mind the (participation) gap: Vouchers, voting, and visibility. USC Law Legal Studies Paper No. 19-9; USC CLASS Research Paper No. CLASS 19-9, SSRN. https://ssrn.com/abstract=3354826. Accessed 5 July 2019.
Wood, A. K., & Grose, C. R. (2019). Campaign finance transparency affects legislators’ election outcomes and behaviors. SSRN. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3236939. Accessed 5 July 2019.
Wood, A. K., & Lewis, D. E. (2017). Agency performance challenges and agency politicization. Journal of Public Administration Research and Theory, 27(4), 581–595.
Yoshinaka, A., & Grose, C. R. (2008). Ideological hedging in uncertain times: inconsistent legislative representation and voter enfranchisement. British Journal of Political Science, 41(4), 765–794.
Young, M., Reksulak, M., & Shughart, W. F. (2001). The political economy of the IRS. Economics and Politics, 13(2), 201–220.
Zvobgo, K. (2019). Human rights versus national interests: Shifting U.S. public attitudes on the international criminal court, International Studies Quarterly. https://docs.wixstatic.com/ugd/dc2664_541a78e9e6d24bd593be35bf5cb933d4.pdf. Accessed 5 July 2019.
Acknowledgements
We would like to thank Joel Sievert, Jeff Jenkins, and the participants in the Causal Inference and American Political Development conference for feedback on this manuscript.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Grose, C.R., Wood, A.K. Randomized experiments by government institutions and American political development. Public Choice 185, 401–413 (2020). https://doi.org/10.1007/s11127-019-00704-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11127-019-00704-5