Skip to main content
SpringerLink
Log in

Comparison of Statistical Signal Detection Methods Within and Across Spontaneous Reporting Databases

  • Original Research Article
  • Published:
Drug Safety Aims and scope Submit manuscript

Abstract

Background

Most pharmacovigilance departments maintain a system to identify adverse drug reactions (ADRs) through analysis of spontaneous reports. The signal detection algorithms (SDAs) and the nature of the reporting databases vary between operators and it is unclear whether any algorithm can be expected to provide good performance in a wide range of environments.

Objective

The objective of this study was to compare the performance of commonly used algorithms across spontaneous reporting databases operated by pharmaceutical companies and national and international pharmacovigilance organisations.

Methods

220 products were chosen and a reference set of ADRs was compiled. Within four company, one national and two international databases, 15 SDAs based on five disproportionality methods were tested. Signals of disproportionate reporting (SDRs) were calculated at monthly intervals and classified by comparison with the reference set. These results were summarised as sensitivity and precision for each algorithm in each database.

Results

Different algorithms performed differently between databases but no method dominated all others. Performance was strongly dependent on the thresholds used to define a statistical signal. However, the different disproportionality statistics did not influence the achievable performance. The relative performance of two algorithms was similar in different databases. Over the lifetime of a product there is a reduction in precision for any method.

Conclusions

In designing signal detection systems, careful consideration should be given to the criteria that are used to define an SDR. The choice of disproportionality statistic does not appreciably affect the achievable range of signal detection performance and so this can primarily be based on ease of implementation, interpretation and minimisation of computing resources. The changes in sensitivity and precision obtainable by replacing one algorithm with another are predictable. However, the absolute performance of a method is specific to the database and is best assessed directly on that database. New methods may be required to gain appreciable improvements.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. To avoid confusion it should be noted that an SDR does not necessarily fulfil the requirements of a signal as defined in pharmacovigilance.

  2. MedDRA® (the Medical Dictionary for Regulatory Activities) terminology is the international medical terminology developed under the auspices of the International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH). MedDRA® trademark is owned by the International Federation of Pharmaceutical Manufacturers and Associations (IFPMA) on behalf of ICH.

References

  1. Bate A, Lindquist M, Edwards IR, Olsson S, Orre R, Lansner A, et al. A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol. 1998;54(4):315–21.

    Article  CAS  PubMed  Google Scholar 

  2. Evans SJW, Waller PC, Davis S. Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf. 2001;10(6):483–6.

    Article  CAS  PubMed  Google Scholar 

  3. Lindquist M, Stahl M, Bate A, Edwards IR, Meyboom RHB. A retrospective evaluation of a data mining approach to aid finding new adverse drug reaction signals in the WHO international database. Drug Saf. 2000;23(6):533–42.

    Article  CAS  PubMed  Google Scholar 

  4. Szarfman A, Machado SG, O’Neill RT. Use of screening algorithms and computer systems to efficiently signal higher-than-expected combinations of drugs and events in the US FDA’s spontaneous reports database. Drug Saf. 2002;25(6):381–92.

    Article  CAS  PubMed  Google Scholar 

  5. Van Puijenbroek EP, Diemont WL, Van Grootheest K. Application of quantitative signal detection in the Dutch spontaneous reporting system for adverse drug reactions. Drug Saf. 2003;26(5):293–301.

    Article  PubMed  Google Scholar 

  6. Almenoff J, Tonning JM, Gould AL, Szarfman A, Hauben M, Ouellet-Hellstrom R, et al. Perspectives on the use of data mining in pharmacovigilance. Drug Saf. 2005;28(11):981–1007.

    Article  CAS  PubMed  Google Scholar 

  7. Roux E, Thiessard F, Fourrier A, Begaud B, Tubert-Bitter P. Evaluation of statistical association measures for the automatic signal generation in pharmacovigilance. IEEE Trans Inf Technol Biomed. 2005;9(4):518–27.

    Article  PubMed  Google Scholar 

  8. Czarnecki A, Voss S. Safety signals using proportional reporting ratios from company and regulatory authority databases. Drug Inf J 2008. 2008;42(3):205–210.

  9. Hauben M, Horn S, Reich L. Potential use of data-mining algorithms for the detection of ‘surprise’ adverse drug reactions. Drug Saf. 2007;30(2):143–55.

    Article  PubMed  Google Scholar 

  10. Bate A, Lindquist M, Edwards IR. The application of knowledge discovery in databases to post-marketing drug safety: example of the WHO database. Fundam Clin Pharmacol. 2008;22(2):127–40.

    Article  CAS  PubMed  Google Scholar 

  11. Banks D, Woo EJ, Burwen DR, Perucci P, Braun MM, Ball R. Comparing data mining methods on the VAERS database. Pharmacoepidemiol Drug Saf. 2005;14(9):601–9.

    Article  PubMed  Google Scholar 

  12. Gipson G. A shrinkage-based comparative assessment of observed-to-expected disproportionality measures. Pharmacoepidemiol Drug Saf. 2012;21(6):589–96.

    Article  PubMed  Google Scholar 

  13. Hauben M, Aronson JK. Defining ‘signal’ and its subtypes in pharmacovigilance based on a systematic review of previous definitions. Drug Saf. 2009;32(2):99–110.

    Article  PubMed  Google Scholar 

  14. Finney DJ. Statistical logic in the monitoring of reactions to therapeutic drugs. Methods Inf Med. 1971;10(4):237–45.

    CAS  PubMed  Google Scholar 

  15. Declerck G, Bousquet C, Jaulent MC. Automatic generation of MedDRA terms groupings using an ontology. Stud Health Technol Inform. 2012;180:73–7.

    PubMed  Google Scholar 

  16. Hochberg AM, Hauben M, Pearson RK, OHara DJ, Reisinger SJ, Goldsmith DI, et al. An evaluation of three signal-detection algorithms using a highly inclusive reference event database. Drug Saf. 2009;32(6):509–25.

    Article  PubMed  Google Scholar 

  17. Slattery J, Alvarez Y, Hidalgo A. Choosing thresholds for statistical signal detection with the proportional reporting ratio. Drug Saf. 2013;36(8):687–92.

    Article  PubMed  Google Scholar 

  18. Gogolak VV. The effect of backgrounds in safety analysis: The impact of comparison cases on what you see. Pharmacoepidemiol Drug Saf. 2003;12(3):249–52.

    Article  PubMed  Google Scholar 

  19. Hammond IW, Gibbs TG, Seifert HA, Rich DS. Database size and power to detect safety signals in pharmacovigilance. Expert Opin Drug Saf. 2007;6(6):713–21.

    Article  CAS  PubMed  Google Scholar 

  20. Hammond IW, Rich DS, Gibbs TG. Effect of consumer reporting on signal detection: Using disproportionality analysis. Expert Opin Drug Saf. 2007;6(6):705–12.

    Article  PubMed  Google Scholar 

  21. Almenoff JS, LaCroix KK, Yuen NA, Fram D, DuMouchel W. Comparative performance of two quantitative safety signalling methods: implications for use in a pharmacovigilance department. Drug Saf. 2006;29(10):875–87.

    Article  CAS  PubMed  Google Scholar 

  22. Brown JS, Petronis K, Bate A, Zhang F, Dashevsky I, Kulldorff M, et al. Comparing two methods for detecting adverse event signals in observational data: empirical Bayes gamma poisson shrinker vs. tree-based scan statistic. Pharmacoepidemiol Drug Saf. 2011;20:S144.

    Google Scholar 

  23. Bunchuailua W, Zuckerman I, Kulsomboon V, Suwankesawong W, Singhasivanon P, Kaewkungwal J. A comparison of signal detection performance between reporting ODDS ratio and Bayesian confidence propagation neural network methods on adverse drug reaction spontaneous reporting database of the Thai FDA. Value Health. 2010;13(7):A508.

    Article  Google Scholar 

  24. Chen Y, Guo JJ, Steinbuch M, Lin X, Buncher CR, Patel NC. Comparison of sensitivity and timing of early signal detection of four frequently used signal detection methods: An empirical study based on the US FDA adverse event reporting system database. Pharm Med. 2008;22(6):359–65.

    Article  Google Scholar 

  25. Harpaz R, Dumouchel W, Lependu P, Bauer-Mehren A, Ryan P, Shah NH. Performance of pharmacovigilance signal-detection algorithms for the FDA adverse event reporting system. Clin Pharmacol Ther. 2013;93(6):539–46.

    Article  CAS  PubMed  Google Scholar 

  26. Van Puijenbroek EP, Bate A, Leufkens HGM, Lindquist M, Orre R, Egberts ACG. A comparison of measures of disproportionality for signal detection is spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiol Drug Saf. 2002;11(1):3–10.

    Article  PubMed  Google Scholar 

  27. Caster O, Noren G, Niklas, Madigan D, Bate A. Large-scale regression-based pattern discovery: the example of screening the WHO global drug safety database. Stat Anal Data Min. 2010;3(4):197–208.

  28. Tatonetti NP, Ye PP, Daneshjou R, Altman RB. Data-driven prediction of drug effects and interactions. Sci Transl Med. 2012;4(125):125ra31.

    Article  PubMed Central  PubMed  Google Scholar 

  29. Kubota K, Koide D, Hirai T. Comparison of data mining methodologies using Japanese spontaneous reports. Pharmacoepidemiol Drug Saf. 2004;13(6):387–94.

    Article  PubMed  Google Scholar 

  30. Kurz X, Slattery J, Addis A, Durand J, Segec A, Skibicka I, et al. The EudraVigilance database of spontaneous adverse reactions as a tool for H1N1 vaccine safety monitoring. Pharmacoepidemiol Drug Saf. 2010;19:S330–1.

    Google Scholar 

  31. Alvarez Y, Hidalgo A, Maignen F, Slattery J. Validation of statistical signal detection procedures in EudraVigilance post-authorization data: a retrospective evaluation of the potential for earlier signalling. Drug Saf. 2010;33(6):475–87.

    Article  PubMed  Google Scholar 

  32. IMI PROTECT. ADR database. http://www.imi-protect.eu/methodsRep.shtml. Accessed 17 Mar 2014.

  33. Clark JA, Klincewicz SL, Stang PE. Spontaneous adverse event signaling methods: classification and use with health care treatment products. Epidemiol Rev. 2001;23(2):191–210.

    Article  CAS  PubMed  Google Scholar 

  34. Norén GN, Hopstadius J, Bate A. Shrinkage observed-to-expected ratios for robust and transparent large-scale pattern discovery. Stat Methods Med Res. 2013;22(1):57–69.

    Article  PubMed  Google Scholar 

  35. DuMouchel W, Pregibon D. Empirical Bayes screening for multi-item associations. In: Proceedings of the seventh ACM SIGKDD international conference on knowledge discovery and data mining. San Francisco: ACM Press; 2001. pp. 67–76.

    Chapter  Google Scholar 

Download references

Acknowledgments

We would like to thank Jenny Wong, Jeffery Painter, Ramin Arani, Niklas Noren, Phil Tregunno, Michael Kayser and Magnus Lerch for their part in the study and constructive comments. The views expressed in this paper are those of the authors only and do not reflect the official policy or position of the IMI JU (Innovative Medicines Initiative Joint Undertaking), the European Union, European Federation of Pharmaceutical Industries and Associations (EFPIA) or the Medicines and Healthcare products Regulatory Agency.

Conflicts of interest

The PROTECT project has received support from the Innovative Medicine Initiative Joint Undertaking (http://www.imi.europa.eu) under Grant Agreement No. 115004, resources of which are composed of financial contribution from the European Union’s Seventh Framework Programme (FP7/2007–2013) and EFPIA companies’ in kind contribution. Gianmario Candore, Kristina Juhlin, Katrin Manlik, Bharat Thakrar, Naashika Quarcoo, Suzie Seabroke, Antoni Wisniewski and Jim Slattery have no financial interest in any commercial signal detection software. Katrin Manlik is an employee of Bayer Pharma AG, Bharat Thakrar is an employee of Roche and holds shares in both Roche and GlaxoSmithKline, Naashika Quarcoo is an employee of and holds shares in GlaxoSmithKline, and Antoni Wisniewski is an employee of and holds shares in AstraZeneca. Products from these companies were among those used to test the methodologies in this research.

Author information

Authors and Affiliations

  1. European Medicines Agency, 7 Westferry Circus, Canary Wharf, London, E14 4HB, UK

    Gianmario Candore & Jim Slattery

  2. Uppsala Monitoring Centre, Uppsala, Sweden

    Kristina Juhlin

  3. Bayer Pharma AG, Berlin, Germany

    Katrin Manlik

  4. Roche, Basel, switzerland

    Bharat Thakrar

  5. GlaxoSmithKline, London, UK

    Naashika Quarcoo

  6. UK Medicines and Healthcare Products Regulatory Agency, London, UK

    Suzie Seabroke

  7. AstraZeneca, Alderley Park, UK

    Antoni Wisniewski

Authors
  1. Gianmario Candore
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kristina Juhlin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Katrin Manlik
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bharat Thakrar
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Naashika Quarcoo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Suzie Seabroke
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Antoni Wisniewski
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jim Slattery
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jim Slattery.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Candore, G., Juhlin, K., Manlik, K. et al. Comparison of Statistical Signal Detection Methods Within and Across Spontaneous Reporting Databases. Drug Saf 38, 577–587 (2015). https://doi.org/10.1007/s40264-015-0289-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40264-015-0289-5

Keywords

Search

Navigation