Comparing public communication in democracies and autocracies: automated text analyses of speeches by heads of government

Abstract

Renewed efforts at empirically distinguishing between different forms of political regimes leave out the cultural dimension. In this article, we demonstrate how modern computational tools can be used to fill this gap. We employ web-scraping techniques to generate a data set of speeches by heads of government in European democracies and autocratic regimes around the globe. Our data set includes 4740 speeches delivered between 1999 and 2019 by 40 political leaders of 27 countries. By scaling the results of a dictionary application, we show how, in comparative terms, liberal or illiberal the leaders present themselves to their national and international audience. In order to gauge whether our liberalness scale reveals meaningful distinctions, we perform a series of validity tests: criterion validity, qualitative hand-coding, unsupervised topic modeling, and network analysis. All tests suggest that our liberalness scale does capture meaningful differences between political regimes despite the large heterogeneity of our data.

Appendix

1.1 Dictionary terms

1.2 Testing the effects of text cleaning procedures

Typically, text-as-data approaches based on the bag-of-words assumption rely on heavy preprocessing of the texts. Different options for preprocessing exist: capital letters can be converted into small letters, stopwords removed, words be lemmed or stemmed etc. Preprocessing decisions can, and often do, influence the results of text analysis. It therefore is paramount to test how robust the results obtained are against equally plausible preprocessing strategies.

Based on our theoretical conceptualization of illiberal and liberal rhetoric in the speeches of heads of government, we choose the following combination of preprocessing steps: Removing punctuation (P), removing numbers (N), put to lowercase (L), stemming (S), remove stopwords (W), and remove infrequently used terms (I). Concerning lowercase, we do not expect huge effects on the results since this is a rather basic procedure. Because all our 4740 speeches were automatically collected from the Internet by using webscraping techniques, we expect most texts to include numbers and other (foreign) signs with no relation to the original text of the speech (e.g. hyperlinks, frames in foreign languages, etc.). This is why we deemed it necessary to erase punctuation and numbers. We also removed infrequently used terms and an individually compiled list of stopwords (e.g. foreign language letters and strange words which we considered not relevant for the analysis).²³

In the following, we investigate the potential of our preprocessing strategy—P–N–L–S–W–I—to significantly affect our results. In essence, the diagnostic procedure suggested by Denny and Spirling tests how much documents ‘move’ depending on the applied preprocessing features. It does so by calculating the so-called preText Score which is the result of comparing pairwise distances between documents for a number of preprocessing specifications (cf. Denny and Spirling 2018). For this, we draw a random sample of not preprocessed documents from our corpus (500, as recommended by Denny and Spirling, p. 185) and perform several diagnostics to measure the potential effects of different (combinations of) preprocessing steps. All operations are done in R with Denny and Spirling’s package preText.²⁴

The two core functions of this package give us, first, a plot illustrated in Fig. 6. It displays the distance scores (preText Scores) for a number of combinations of preprocessing steps. Higher scores mean higher effects on results. The plot shows that our chosen combination of preprocessing features—P–N–L–S–W–I, marked with a green line—is in the medium range if compared to other preprocessing specifications. This indicates that our chosen text preprocessing can be expected to have a comparatively not so big effect on the results.

Fig. 6

Assessing the expected effects of our P–N–L–S–W–I preprocessing procedure on our corpus’ analysis

A second plot, Fig. 7, shows regression coefficients for each single text cleaning feature. Here negative coefficients indicate that a step tends to reduce the unusualness of the results, positive coefficients indicate that applying the step is likely to produce more unusual results for our corpus. The plot shows that it is particularly the feature of removing stopwords which has high potential to reduce the unusualness of our results and the removing of punctuation that has high potential to increase the unusualness of our results. Yet, as explained above, we assume that this unusualness in the not yet preprocessed documents is caused by the web-scraping procedure, thus consists of frequent foreign language letters, word fragments and punctuation which are not relevant for our analysis of illiberal and liberal language styles. That is why we can safely remove such stopwords and punctuation during preprocessing despite the expected effect.

Overall and based on these diagnostics and robustness tests we can conclude that our combined text cleaning procedures have relatively low effects on the results of our corpus’ analysis and that based on our preprocessing theory, we can accept the expectedly high effects of removing stopwords for our corpus.

Fig. 7

Assessing the expected effects of single preprocessing features on our corpus’ analysis (Denny and Spirling 2018)

1.3 Additional material for a 14-topic STM

See Figs. 8, 9 and 10.

Fig. 8

The coefficients and confidence intervals in each plot show the estimated proportions of selected topics in all speeches per speaker, PART I

Fig. 9

The coefficients and confidence intervals in each plot show the estimated proportions of selected topics in all speeches per speaker, PART II

Fig. 10

The coefficients and confidence intervals in each plot show the estimated proportions of selected topics in all speeches per speaker, PART III