Skip to main content
SpringerLink
Log in

Healthcare Funding Decisions and Real-World Benefits: Reducing Bias by Matching Untreated Patients

  • Practical Application
  • Published:
PharmacoEconomics Aims and scope Submit manuscript

Abstract

Governments and health insurers often make funding decisions based on health gains from randomised controlled trials. These decisions are inherently uncertain because health gains in trials may not translate to practice owing to differences in the population, treatment use and setting. Post-market analysis of real-world data can provide additional evidence but estimates from standard matching methods may be biased when unobserved characteristics explain whether a patient is treated and their outcomes. We propose a new untreated matching approach that can reduce this bias. Our approach utilises the outcomes of contemporaneous untreated patients to improve the matching of treated and historical control patients. We assess the performance of this new approach compared to standard matching using a simulation study and demonstrate the steps required using a funding decision for prostate cancer treatments in Australia. Our simulation study shows that our new matching approach eliminates nearly all bias when unobserved treatment selection is related to outcomes, and outperforms standard matching in most scenarios. In our empirical example, standard matching overestimated survival by 15% (95% confidence interval 2–34) compared to our untreated matching approach. The health gains estimated using our approach were slightly lower than expected based on the trial evidence, but we also found evidence that in practice prescribers ceased prior therapies earlier, treated a more vulnerable population and continued treatment for longer. Our untreated matching approach offers researchers a new tool for reducing uncertainty in healthcare funding decisions using real-world data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. Usual care could include no treatment or a range of other active therapies prescribed at different stages in the clinical management pathway before new treatment became available.

  2. Here, we would require the random-like allocation of patients to doctors, e.g. the quick adoption of the new treatment being the only difference in treatment practice. The method we develop in the current paper can be used for the case where the funding decision (instrument) is clearly identified and there is a switch or difference from ‘no use’ of a particular treatment pathway to ‘some use’ of a treatment pathway either across time or space.

  3. Specifically, t0 in clinical trials is the day of randomisation when exposure to the intervention is decided, which may be the same as, or just prior to initiation of treatment.

  4. Immortal-time or survivor bias is an extreme example where matched treated-controls die before they even would have had the opportunity to be treated [26].

  5. Outcomes for censored patients must be simulated before matching on outcomes.

  6. In our approach, some uncertainty comes from whether there are too many or too few true untreated-controls in the matched pre-funding cohort. Too many may inflate the estimated treatment effect, whereas too few may underestimate the effect, given untreated-controls may be expected to have worse outcomes in general than treated-controls.

  7. The instrumental variables’ independence assumption is E(H|X,D) = E(H|X), where H is the health outcome, X is whether (X = 1) or not (X = 0) the individual was treated, and D is whether (D = 1) or not (D = 0) the treatment was available (i.e. the instrument). This assumption implies that E(H|X = 0, D = 0) = E(H|X = 0, D = 1) or that the instrument does not affect health outcomes for those that were unaffected by the funding decision.

  8. For completeness, we also identified SM treated-controls from the untreated patients in the post-funding period, which results in a relative bias of 76% in the base case. This illustrates that with treatment selection using contemporaneous controls can produce very biased results.

  9. The true average treatment effect for the treated was estimated as the average survival for the treated patient (with treatment) compared to their counterfactual survival (without treatment), using a sample size of 10,000,000.

  10. In this scenario, UM did not fail on any of the iterations because the proportion of untreated patients in each disease category (Z) was relatively low and the CEM bins were relatively coarse (i.e. there were many potential controls).

  11. One trial used an active comparator but is equivalent to placebo for survival given the active comparator was known not to improve survival [33].

  12. Compared to clinical practice in clinical trials there can be a delay between becoming eligible and starting treatment because of the trial recruitment protocols.

  13. We estimated the treatment effect for SM using historical controls, which is consistent with the approach taken in Sect. 4. In addition, there were not many potential contemporaneous controls in this example given a large proportion (87.8%) received further treatment after the funding decision.

  14. The registries exclude experimental treatment, but this was unlikely to significantly bias results. Australian patients included in this analysis were unlikely to have been included in prostate cancer trials during the sample period given the trial enrolment dates and eligibility criteria [42,43,44].

  15. We predicted survival for censored patients (we observed time on docetaxel for all patients).

  16. Most untreated patients died shortly after ceasing docetaxel, whereas treated patients initiated one of the new treatments.

  17. For completeness, we also quality adjusted the estimated survival gains using utility estimates from the literature [51,52,53,54,55,56,57] (see the ESM for methods and results).

  18. The trials excluded sicker patients with multiple comorbidities and a short life expectancy, and had strict stopping criteria [33, 39]. As the new treatments were not available prior to the trials, many trial patients had already started to receive further ‘palliative’ chemotherapy (that is, chemotherapy without evidence of improved survival).

  19. Note that while the 95% CIs of the estimated treatment effects from SM and UM largely overlap, they are highly dependent with each other such that the 95% CIs for the difference between the SM and UM estimates are significant. We can see this in Fig. A.9 of the ESM where for each bootstrapped sample the SM estimate is nearly always larger than the UM estimate.

  20. Based on the trial data and a crude assumption of additive survival gains for second-line and third-line treatment, the expected survival gain is ≥6 months. At censoring, 37% of patients had received two treatments but we expect a higher proportion when extrapolated corresponding to a slightly longer expected survival (i.e. 4.3 months + 0.37*4.3 months = 5.9 months). It should be noted that the impact of the funding decision may not only include the impacts of the new treatments themselves, but how their use may change the treatment pathway for those individuals (such as less time on prior treatment).

References

  1. Stafinski T, McCabe CJ, Menon D. Funding the unfundable. Pharmacoeconomics. 2010;28(2):113–42.

    Article  PubMed  Google Scholar 

  2. Nordon C, Karcher H, Groenwold RH, Ankarfeldt MZ, Pichler F, Chevrou-Severac H, et al. The “efficacy-effectiveness gap”: historical background and current conceptualization. Value Health. 2016;19(1):75–81.

    Article  PubMed  Google Scholar 

  3. Ghijben P, Gu Y, Lancsar E, Zavarsek S. Revealed and stated preferences of decision makers for priority setting in health technology assessment: a systematic review. Pharmacoeconomics. 2018;36(3):323–40.

    Article  PubMed  Google Scholar 

  4. Parkinson B, Sermet C, Clement F, Crausaz S, Godman B, Garner S, et al. Disinvestment and value-based purchasing strategies for pharmaceuticals: an international review. Pharmacoeconomics. 2015;33(9):905–24.

    Article  PubMed  Google Scholar 

  5. Tuffaha HW, Scuffham PA. The Australian managed entry scheme: are we getting it right? Pharmacoeconomics. 2018;36(5):555–65.

    Article  PubMed  Google Scholar 

  6. Makady A, van Veelen A, Jonsson P, Moseley O, D’Andon A, de Boer A, et al. Using real-world data in health technology assessment (HTA) practice: a comparative study of five HTA agencies. Pharmacoeconomics. 2018;36(3):359–68.

    Article  PubMed  Google Scholar 

  7. Berger ML, Sox H, Willke RJ, Brixner DL, Eichler HG, Goettsch W, et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: recommendations from the joint ISPOR-ISPE Special Task Force on Real-World Evidence in Health Care Decision Making. Pharmacoepidemiol Drug Saf. 2017;26(9):1033–9.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Visvanathan K, Levit LA, Raghavan D, Hudis CA, Wong S, Dueck A, et al. Untapped potential of observational research to inform clinical decision making: American Society of Clinical Oncology research statement. J Clin Oncol. 2017;35(16):1845–54.

    Article  PubMed  Google Scholar 

  9. Velentgas P, Dreyer NA, Nourjah P, Smith SR, Torchia MM. Developing a protocol for observational comparative effectiveness research: a user's guide: AHRQ Publication No. 12(13)-EHC099. Rochville (MD): Agency for Health Research and Quality; Jan 2013.

  10. Clarke GM, Conti S, Wolters AT, Steventon A. Evaluating the impact of healthcare interventions using routine data. BMJ. 2019;365:l2239.

    Article  PubMed  PubMed Central  Google Scholar 

  11. de Pouvourville G, Blin P, Karam P. The contribution of real-world evidence to cost-effectiveness analysis: case study of dabigatran etexilate in France. Eur J Health Econ. 2020;21(2):235–49.

    Article  PubMed  Google Scholar 

  12. Schilling C, Petrie D, Dowsey MM, Choong PF, Clarke P. The impact of regression to the mean on economic evaluation in quasi-experimental pre–post studies: the example of total knee replacement using data from the Osteoarthritis Initiative. Health Econ. 2017;26(12):e35-51.

    Article  PubMed  Google Scholar 

  13. Johar M. The impact of the Indonesian health card program: a matching estimator approach. J Health Econ. 2009;28(1):35–53.

    Article  PubMed  Google Scholar 

  14. The European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP). Guide on methodological standards in pharmacoepidemiology (Revision 6). EMA/95098/2010. London: European Medicines Agency; 2018.

  15. Rothwell PM. Factors that can affect the external validity of randomised controlled trials. PLoS Clin Trials. 2006;1(1):e9.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Kennedy-Martin T, Curtis S, Faries D, Robinson S, Johnston J. A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results. Trials. 2015;16(1):495.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Booth C, Tannock I. Randomised controlled trials and population-based observational research: partners in the evolution of medical evidence. Br J Cancer. 2014;110(3):551.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Zarin DA, Young JL, West JC. Challenges to evidence-based medicine. Soc Psychiatry Psychiatr Epidemiol. 2005;40(1):27–35.

    Article  PubMed  Google Scholar 

  19. Treweek S, Zwarenstein M. Making trials matter: pragmatic and explanatory trials and the problem of applicability. Trials. 2009;10(1):37.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Parkinson B, Viney R, Haas M, Goodall S, Srasuebkul P, Pearson S-A. Real-world evidence: a comparison of The Australian Herceptin Program and clinical trials of trastuzumab for HER2-positive metastatic breast cancer. Pharmacoeconomics. 2016;34(10):1039–50.

    Article  PubMed  Google Scholar 

  21. World Medical Association. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA. 2013;310(20):2191–4.

    Article  Google Scholar 

  22. Tuffaha HW, Gordon LG, Scuffham PA. Value of information analysis in healthcare: a review of principles and applications. J Med Econ. 2014;17(6):377–83.

    Article  PubMed  Google Scholar 

  23. Fleurence RL, Naci H, Jansen JP. The critical role of observational evidence in comparative effectiveness research. Health Aff. 2010;29(10):1826–33.

    Article  Google Scholar 

  24. Rubin DB. Causal inference using potential outcomes: design, modeling, decisions. J Am Stat Assoc. 2005;100(469):322–31.

    Article  CAS  Google Scholar 

  25. Rochon PA, Gurwitz JH, Sykora K, Mamdani M, Streiner DL, Garfinkel S, et al. Reader’s guide to critical appraisal of cohort studies: 1. Role and design. BMJ. 2005;330(7496):895–7.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Suissa S. Immortal time bias in observational studies of drug effects. Pharmacoepidemiol Drug Saf. 2007;16(3):241–9.

    Article  PubMed  Google Scholar 

  27. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70(1):41–55.

    Article  Google Scholar 

  28. Iacus SM, King G, Porro G. Causal inference without balance checking: coarsened exact matching. Polit Anal. 2012;20(1):1–24.

    Article  Google Scholar 

  29. Sheetz KH, Norton EC, Regenbogen SE, Dimick JB. An instrumental variable analysis comparing medicare expenditures for laparoscopic vs open colectomy. JAMA Surg. 2017;152(10):921–9.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Cain LE, Cole SR, Greenland S, Brown TT, Chmiel JS, Kingsley L, et al. Effect of highly active antiretroviral therapy on incident AIDS using calendar period as an instrumental variable. Am J Epidemiol. 2009;169(9):1124–32.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Rascati KL, Johnsrud MT, Crismon ML, Lage MJ, Barber BL. Olanzapine versus risperidone in the treatment of schizophrenia. Pharmacoeconomics. 2003;21(10):683–97.

    Article  PubMed  Google Scholar 

  32. Tchetgen EJT, Walter S, Vansteelandt S, Martinussen T, Glymour M. Instrumental variable estimation in a survival context. Epidemiology. 2015;26(3):402–10.

    Article  Google Scholar 

  33. De Bono JS, Oudard S, Ozguroglu M, Hansen S, Machiels J-P, Kocak I, et al. Prednisone plus cabazitaxel or mitoxantrone for metastatic castration-resistant prostate cancer progressing after docetaxel treatment: a randomised open-label trial. Lancet. 2010;376(9747):1147–54.

    Article  PubMed  Google Scholar 

  34. Australian Government. Public summary document. Cabazitaxel, July 2011. Pharmaceutical Benefits Advisory Committee Meeting, 6–8 July 2011, Sydney; 2011.

  35. Australian Government. Public summary document. Abiraterone, November 2011. Pharmaceutical Benefits Advisory Committee Meeting, 2–4 November 2011, Melbourne; 2011.

  36. Australian Government. Public summary document. Enzalutamide, July 2014. Pharmaceutical Benefits Advisory Committee Meeting, 9–11 July 2014, Canberra; 2014.

  37. Australian Government. Public summary document. Abiraterone, November 2012. Pharmaceutical Benefits Advisory Committee Meeting, 7–9 November 2012, Canberra; 2012.

  38. Bahl A, Oudard S, Tombal B, Özgüroĝlu M, Hansen S, Kocak I, et al. Impact of cabazitaxel on 2-year survival and palliation of tumour-related pain in men with metastatic castration-resistant prostate cancer treated in the TROPIC trial. Ann Oncol. 2013;24(9):2402–8.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  39. De Bono JS, Logothetis CJ, Molina A, Fizazi K, North S, Chu L, et al. Abiraterone and increased survival in metastatic prostate cancer. N Engl J Med. 2011;364(21):1995–2005.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Fizazi K, Scher HI, Molina A, Logothetis CJ, Chi KN, Jones RJ, et al. Abiraterone acetate for treatment of metastatic castration-resistant prostate cancer: final overall survival analysis of the COU-AA-301 randomised, double-blind, placebo-controlled phase 3 study. Lancet Oncol. 2012;13(10):983–92.

    Article  CAS  PubMed  Google Scholar 

  41. Scher HI, Fizazi K, Saad F, Taplin M-E, Sternberg CN, Miller K, et al. Increased survival with enzalutamide in prostate cancer after chemotherapy. N Engl J Med. 2012;367(13):1187–97.

    Article  CAS  PubMed  Google Scholar 

  42. Beer TM, Armstrong AJ, Rathkopf DE, Loriot Y, Sternberg CN, Higano CS, et al. Enzalutamide in metastatic prostate cancer before chemotherapy. N Engl J Med. 2014;371(5):424–33.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Ryan CJ, Smith MR, De Bono JS, Molina A, Logothetis CJ, De Souza P, et al. Abiraterone in metastatic prostate cancer without previous chemotherapy. N Engl J Med. 2013;368(2):138–48.

    Article  CAS  PubMed  Google Scholar 

  44. Sternberg CN, Castellano D, Daugaard G, Géczi L, Hotte SJ, Mainwaring PN, et al. Abiraterone acetate for patients with metastatic castration-resistant prostate cancer progressing after chemotherapy: final analysis of a multicentre, open-label, early-access protocol trial. Lancet Oncol. 2014;15(11):1263–8.

    Article  CAS  PubMed  Google Scholar 

  45. Sarfati D, Gurney J, Stanley J, Lim BT, McSherry C. Development of a pharmacy-based comorbidity index for patients with cancer. Med Care. 2014;52(7):586–93.

    Article  PubMed  Google Scholar 

  46. Moreira DM, Howard LE, Sourbeer KN, Amarasekara HS, Chow LC, Cockrell DC, et al. Predicting time from metastasis to overall survival in castration-resistant prostate cancer: results from SEARCH. Clin Genitourin Cancer. 2017;15(1):60-6.e2.

    Article  PubMed  Google Scholar 

  47. Armstrong AJ, Garrett-Mayer ES, Yang Y-CO, de Wit R, Tannock IF, Eisenberger M. A contemporary prognostic nomogram for men with hormone-refractory metastatic prostate cancer: a TAX327 study analysis. Clin Cancer Res. 2007;13(21):6396–403.

    Article  CAS  PubMed  Google Scholar 

  48. Heidenreich A, Bastian PJ, Bellmunt J, Bolla M, Joniau S, van der Kwast T, et al. EAU guidelines on prostate cancer. Part II: treatment of advanced, relapsing, and castration-resistant prostate cancer. Eur Urol. 2014;65(2):467–79.

    Article  CAS  PubMed  Google Scholar 

  49. Horwich A, Parker C, de Reijke T, Kataja V, ESMO Guidelines Working Group. Prostate cancer: ESMO clinical practice guidelines for diagnosis, treatment and follow-up. Ann Oncol. 2013;24(Suppl. 6):vi106–14.

    Article  PubMed  Google Scholar 

  50. Horwich A, Parker C, Bangma C, Kataja V, ESMO Guidelines Working Group. Prostate cancer: ESMO clinical practice guidelines for diagnosis, treatment and follow-up. Ann Oncol. 2010;21(Suppl. 5):v129–33.

    Article  PubMed  Google Scholar 

  51. Australian Government. Public summary document. Enzalutamide, November 2015. Pharmaceutical Benefits Advisory Committee Meeting, 4–6 November 2015, Canberra; 2015.

  52. Sandblom G, Carlsson P, Sennfält K, Varenhorst E. A population-based study of pain and quality of life during the year before death in men with prostate cancer. Br J Cancer. 2004;90(6):1163.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  53. Wolff J, Donatz V, Klier J, Erhardt W, Dass R, Geiges G. PCN120 quality of life among German patients with metastatic castration-resistant prostate cancer. Value Health. 2012;15(7):A431.

    Article  Google Scholar 

  54. Diels J, Hamberg P, Ford D, Price PW, Spencer M, Dass R. Mapping FACT-P to EQ-5D in a large cross-sectional study of metastatic castration-resistant prostate cancer patients. Qual Life Res. 2015;24(3):591–8.

    Article  CAS  PubMed  Google Scholar 

  55. Färkkilä N, Torvinen S, Roine RP, Sintonen H, Hänninen J, Taari K, et al. Health-related quality of life among breast, prostate, and colorectal cancer patients with end-stage disease. Qual Life Res. 2014;23(4):1387–94.

    Article  PubMed  Google Scholar 

  56. Lloyd AJ, Kerr C, Penton J, Knerer G. Health-related quality of life and health utilities in metastatic castrate-resistant prostate cancer: a survey capturing experiences from a diverse sample of UK patients. Value Health. 2015;18(8):1152–7.

    Article  PubMed  Google Scholar 

  57. National Institute for Health Clinical Excellence. Technology appraisal guidance 259: Abiraterone for castration-resistant metastatic prostate cancer previously treated with a docetaxel-containing regimen. London: National Institute for Health and Clinical Excellence; 2012.

    Google Scholar 

  58. Efron B, Tibshirani RJ. An Introduction to the Bootstrap. New York: Chapman & Hall, 1994.

    Book  Google Scholar 

Download references

Acknowledgements

The authors are grateful to the Department of Human Services, Australian Government, for providing access to the data used in this paper. The findings and views of this paper are those of the authors and are not attributed to the Department of Human Services. This paper benefited from discussion with seminar participants at Monash University, University of Queensland, University of Adelaide, University of Western Australia, and the Australian Health Economics Society Conference 2019. We would also like to thank Duncan Mortimer, Jon Karnon, Johannes Kunz and two anonymous referees who provided helpful comments on previous drafts of this paper.

Author information

Authors and Affiliations

  1. Centre for Health Economics, Monash Business School, Monash University, Caulfield East, VIC, Australia

    Peter Ghijben, Dennis Petrie & Gang Chen

  2. Deakin Health Economics, Centre for Population Health Research, Deakin University, Geelong, VIC, Australia

    Silva Zavarsek

  3. Department of Health Services Research and Policy, College of Health and Medicine, The Australian National University, Acton, ACT, Australia

    Emily Lancsar

Authors
  1. Peter Ghijben
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dennis Petrie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Silva Zavarsek
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Emily Lancsar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peter Ghijben.

Ethics declarations

Funding

The Centre for Health Economics, Monash University funded this paper.

Conflicts of interest/Competing interests

Peter Ghijben, Dennis Petrie, Silva Zavarsek, Gang Chen, and Emily Lancsar have no conflicts of interest that are directly relevant to the content of this article.

Ethics approval

Monash University Human Research Ethics Committee granted ethics approval for this study (CF15/4240-2015001816).

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of Data and Material

This paper uses administrative patient-level data owned by the Australian Government. The data are not publicly available because they contain confidential health records on Australian patients. Access to the data requires a formal application to ‘Services Australia’.

Code availability

The Stata code for the Monte Carlo simulations is provided in the ESM.

Authors’ contributions

All authors contributed to the study conception and design. Material preparation and data collection were performed by PG and SZ. The methodology was developed by PG and DP, and analysis was performed by PG. The first draft of the manuscript was written by PG and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 11299 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ghijben, P., Petrie, D., Zavarsek, S. et al. Healthcare Funding Decisions and Real-World Benefits: Reducing Bias by Matching Untreated Patients. PharmacoEconomics 39, 741–756 (2021). https://doi.org/10.1007/s40273-021-01020-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40273-021-01020-x

Search

Navigation