Skip to main content
SpringerLink
Log in

Alternative Outcome Definitions and Their Effect on the Performance of Methods for Observational Outcome Studies

  • Original Research Article
  • Published:
Drug Safety Aims and scope Submit manuscript

Abstract

Background

A systematic risk identification system has the potential to test marketed drugs for important Health Outcomes of Interest or HOI. For each HOI, multiple definitions are used in the literature, and some of them are validated for certain databases. However, little is known about the effect of different definitions on the ability of methods to estimate their association with medical products.

Objectives

Alternative definitions of HOI were studied for their effect on the performance of analytical methods in observational outcome studies.

Methods

A set of alternative definitions for three HOI were defined based on literature review and clinical diagnosis guidelines: acute kidney injury, acute liver injury and acute myocardial infarction. The definitions varied by the choice of diagnostic codes and the inclusion of procedure codes and lab values. They were then used to empirically study an array of analytical methods with various analytical choices in four observational healthcare databases. The methods were executed against predefined drug-HOI pairs to generate an effect estimate and standard error for each pair. These test cases included positive controls (active ingredients with evidence to suspect a positive association with the outcome) and negative controls (active ingredients with no evidence to expect an effect on the outcome). Three different performance metrics where used: (i) Area Under the Receiver Operator Characteristics (ROC) curve (AUC) as a measure of a method’s ability to distinguish between positive and negative test cases, (ii) Measure of bias by estimation of distribution of observed effect estimates for the negative test pairs where the true effect can be assumed to be one (no relative risk), and (iii) Minimal Detectable Relative Risk (MDRR) as a measure of whether there is sufficient power to generate effect estimates.

Results

In the three outcomes studied, different definitions of outcomes show comparable ability to differentiate true from false control cases (AUC) and a similar bias estimation. However, broader definitions generating larger outcome cohorts allowed more drugs to be studied with sufficient statistical power.

Conclusions

Broader definitions are preferred since they allow studying drugs with lower prevalence than the more precise or narrow definitions while showing comparable performance characteristics in differentiation of signal vs. no signal as well as effect size estimation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Barron BA. The effects of misclassification on the estimation of relative risk. Biometrics. 1977;33(2):414–8.

    Article  PubMed  CAS  Google Scholar 

  2. Stang PE, Ryan PB, Racoosin JA, Overhage JM, Hartzema AG, Reich C, et al. Advancing the science for active surveillance: rationale and design for the Observational Medical Outcomes Partnership. Ann Int Med. 2010;153(9):600–6.

    Article  PubMed  Google Scholar 

  3. Carnahan RM, Moores KG. Mini-Sentinel’s systematic reviews of validated methods for identifying health outcomes using administrative and claims data: methods and lessons learned. Pharmacoepidemiol Drug Saf. 2012;21(Suppl 1):82–9.

    Article  PubMed  Google Scholar 

  4. Stang PE, Ryan PB, Dusetzina SB, Hartzema AG, Reich C, Overhage JM, et al. Health outcomes of interest in observational data: issues in identifying definitions in the literature. Health Outcomes Res Med. 2012;3(1):e37–44.

    Article  Google Scholar 

  5. Kellum JA, Bellomo R, Ronco C. Definition and classification of acute kidney injury. Nephron Clinical Pract. 2008;109(4):c182–7.

    Article  Google Scholar 

  6. James M, Pannu N. Methodological considerations for observational studies of acute kidney injury using existing data sources. J Nephrol. 2009;22(3):295–305.

    PubMed  CAS  Google Scholar 

  7. Katz AJ, Ryan PB, Racoosin JA, Stang PE. Assessment of case definitions for identifying acute liver injury in large observational databases. Drug Saf. 2013;36(8):651–61.

    Article  PubMed  Google Scholar 

  8. Ryan PB, Stang PE, Overhage JM, Suchard MA, Hartzema AG, DuMouchel W, et al. A comparison of the empirical performance of methods for a risk identification system. Drug Saf (in this supplement issue). doi:10.1007/s40264-013-0108-9.

  9. Observational Medical Outcomes Partnership Methods Library; 2012. http://omop.org/HOI [cited 2012 December 13].

  10. Overhage JM, Ryan PB, Schuemie MJ, Stang PE. Desideratum for Evidence Based Epidemiology. Drug Saf (in this supplement issue). doi:10.1007/s40264-013-0102-2

  11. Ryan PB, Schuemie MJ, Welebob E, Duke J, Valentine S, Hartzema AG. Defining a reference set to support methodological research in drug safety. Drug Saf (in this supplement issue). doi:10.1007/s40264-013-0097-8

  12. Armstrong B. A simple estimator of minimum detectable relative risk, sample size, or power in cohort studies. Am J Epidemiol. 1987;126(2):356–8.

    Article  PubMed  CAS  Google Scholar 

  13. Reich C, Ryan PB, Suchard MA. The impact of drug and outcome prevalence on the feasibility and performance of analytical methods for a risk identification and analysis system. Drug Saf (in this supplement issue). doi:10.1007/s40264-013-0112-0

  14. Cantor SB, Kattan MW. Determining the area under the ROC curve for a binary diagnostic test. Med Decis Making. 2000;20(4):468–70.

    Article  PubMed  CAS  Google Scholar 

  15. Copeland KT, Checkoway H, McMichael AJ, Holbrook RH. Bias due to misclassification in the estimation of relative risk. Am J Epidemiol. 1977;105(5):488–95.

    PubMed  CAS  Google Scholar 

  16. Wacholder S, Armstrong B, Hartge P. Validation studies using an alloyed gold standard. Am J Epidemiol. 1993;137(11):1251–8.

    PubMed  CAS  Google Scholar 

  17. Evans JM, MacDonald TM. Misclassification and selection bias in case-control studies using an automated database. Pharmacoepidemiol Drug Saf. 1997;6(5):313–8.

    Article  PubMed  CAS  Google Scholar 

  18. Mukherjee D, Nissen SE, Topol EJ. Risk of cardiovascular events associated with selective COX-2 inhibitors. JAMA. 2001;286(8):954–9.

    Article  PubMed  CAS  Google Scholar 

Download references

Acknowledgments

The Observational Medical Outcomes Partnership is funded by the Foundation for the National Institutes of Health (FNIH) through generous contributions from the following: Abbott, Amgen, AstraZeneca, Bayer Healthcare Pharmaceuticals, Biogen Idec, Bristol-Myers Squibb, Eli Lilly & Company, GlaxoSmithKline, Janssen Research and Development, Lundbeck, Merck & Co., Novartis Pharmaceuticals Corporation, Pfizer, Pharmaceutical Research Manufacturers of America (PhRMA), Roche, Sanofi-Aventis, Schering-Plough Corporation, and Takeda. Dr. Reich is an employee of AstraZeneca. Drs. Ryan and Schuemie are employees of Janssen Research and Development. Dr. Schuemie received a fellowship from the Office of Medical Policy, Center for Drug Evaluation and Research, Food and Drug Administration. Dr. Schuemie has previously received a grant from FNIH.

This article was published in a supplement sponsored by the Foundation for the National Institutes of Health (FNIH). The supplement was guest edited by Stephen J.W. Evans. It was peer reviewed by Olaf H. Klungel who received a small honorarium to cover out-of-pocket expenses. S.J.W.E has received travel funding from the FNIH to travel to the OMOP symposium and received a fee from FNIH for the review of a protocol for OMOP. O.H.K has received funding for the IMI-PROTECT project from the Innovative Medicines Initiative Joint Undertaking (http://www.imi.europa.eu) under Grant Agreement no 115004, resources of which are composed of financial contribution from the European Union’s Seventh Framework Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.

Author information

Authors and Affiliations

  1. AstraZeneca PLC, 35 Gatehouse Drive, Waltham, MA, 02451, USA

    Christian G. Reich

  2. Janssen Research and Development LLC, Titusville, NJ, USA

    Patrick B. Ryan

  3. Department of Medical Informatics, Erasmus University Medical Center Rotterdam, Rotterdam, The Netherlands

    Martijn J. Schuemie

  4. Observational Medical Outcomes Partnership, Foundation for the National Institutes of Health, Bethesda, MD, USA

    Christian G. Reich, Patrick B. Ryan & Martijn J. Schuemie

Authors
  1. Christian G. Reich
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Patrick B. Ryan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Martijn J. Schuemie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christian G. Reich.

Additional information

The OMOP research used data from Truven Health Analytics (formerly the Health Business of Thomson Reuters), and includes MarketScan® Research Databases, represented with MarketScan Lab Supplemental (MSLR, 1.2 m persons), MarketScan Medicare Supplemental Beneficiaries (MDCR, 4.6 m persons), MarketScan Multi-State Medicaid (MDCD, 10.8 m persons), MarketScan Commercial Claims and Encounters (CCAE, 46.5 m persons). Data also provided by Quintiles® Practice Research Database (formerly General Electric’s Electronic Health Record, 11.2 m persons) database. GE is an electronic health record database while the other four databases contain administrative claims data.

Appendix: Health Outcome of Interest Definitions Used in the Study

Appendix: Health Outcome of Interest Definitions Used in the Study

HOI

#

Definition

Acute liver injury

1

Occurrence of at least one diagnostic code ICD-9-CM:

• 277.4 ‘Disorders of bilirubin excretion’

• 570* ‘Acute and subacute necrosis of the liver’a

• 572.2 ‘Hepatic coma (hepatorenal syndrome)’

• 572.4* ‘Hepatorenal syndrome’a

• 573* ‘Other disorders of the liver, including chemical or drug induced’a

• 576.8 ‘Other specified disorders of biliary tract’

• 782.4 ‘Jaundice, unspecified, not of newborn’

• 789.1* ‘Hepatomegaly’a

• 790.4* ‘Nonspecific elevation of transaminase or lactic dehydrogenase levels’a

• 794.8* ‘Abnormal liver function test results’a

2

Occurrence of at least one diagnostic code ICD-9-CM:

• 570* ‘Acute and subacute necrosis of the liver’a

• 572.2 ‘Hepatic coma (hepatorenal syndrome)’

• 572.4* ‘Hepatorenal syndrome’a

• 573* ‘Other disorders of the liver, including chemical or drug induced’a

3

Definition Acute Liver Injury #2 AND

Occurrence of at least one diagnostic procedure code within 30 days prior to diagnostic codea AND

Occurrence of at least one therapeutic procedure code within 60 days after the diagnostic codeb

4

Definition Acute Liver Injury #2 AND

Occurrence of at least one diagnostic procedure code within 30 days prior to diagnostic codeb AND

Occurrence of at least one therapeutic procedure code within 60 days after the diagnostic codeb AND

Definition Acute Liver Injury #5

5

Indicative of Hy’s law: Occurrence of the following lab tests and test results within 7 day period:

 Alanine aminotransferase ≥3 times the upper limit of normal as defined in data source, or 40 IU/L if not available OR

 Aspartate aminotransferase ≥3 times the upper limit of normal as defined in data source, or 40 IU/L if not available} AND

Total bilirubin ≥2 times the upper limit of normal as defined in data source, or 1.2 mg/dL if not available

6

Strongly indicative of Hy’s law: Occurrence of the following lab tests and test results within 7 day period:

 Alanine aminotransferase ≥10 times the upper limit of normal as defined in data source, or 40 IU/L if not available OR

 Aspartate aminotransferase ≥10 times the upper limit of normal as defined in data source, or 40 IU/L if not available AND

Total bilirubin ≥2 times the upper limit of normal as defined in data source, or 1.2 mg/dL if not available

7

Occurrence of at least one diagnostic code ICD-9-CM:

• 570* ‘Acute and subacute necrosis of the liver’a

• 572.4* ‘Hepatorenal syndrome’a

• 573.0 ‘Chronic passive congestion of liver’

• 573.1 ‘Hepatitis in viral diseases classified elsewhere’

• 573.4 ‘Hepatic infarction’

8

Definition of Acute Liver Failure #2 AND

Hospitalization at date of diagnostic code

Acute kidney injury

1

Occurrence of at least one diagnostic code ICD-9-CM:

• 584* ‘Acute renal failure’a

2

Definition Acute Kidney Injury #1 AND

Occurrence of at least one therapeutic ICD-9 Procedure code within 60 days after diagnostic code:

• 39.95 ‘Hemodialysis’

• 54.98 ‘Peritoneal dialysis’ AND

Excluding any diagnostic code for chronic dialysis status anytime before the diagnostic code ICD-9-CM:

• V45.1 ‘Renal dialysis status’

• V56.0 ‘Encounter for dialysis and dialysis catheter care’

• V56.31 ‘Encounter for adequacy testing for hemodialysis’

• V56.32 ‘Encounter for adequacy testing for peritoneal dialysis’

• V56.8 ‘Other dialysis’

3

Occurrence of the lab test code LOINC 2160-0 and the following test results:

• Serum creatinine ≥0.5 mg/dL for patients with a baseline level of ≤1.9 mg/dL

• Serum creatinine ≥1.0 mg/dL for patients with a baseline level of 2.0–4.9 mg/dL

• Serum creatinine ≥1.5 mg/dL for patients with a baseline level ≥5.0 mg/dL

The baseline level is defined as the occurrence of most recent lab test result any time prior to the elevated test result

4

Occurrence of at least one diagnostic code ICD-9-CM:

• 584 ‘Acute renal failure’ 584.5 ‘Acute renal failure with lesion of tubular necrosis’

• 584.6 ‘Acute renal failure with lesion of renal cortical necrosis’

• 584.7 ‘Acute renal failure with lesion of renal medullary (papillary) necrosis’

5

Definition of Acute Kidney Injury #1 AND Hospitalization at date of diagnostic code

Acute myocardial infarction

1

Occurrence of at least one broad diagnostic code ICD-9-CM:

• 410* ‘Acute myocardial infarction’a

• 411.1 ‘Intermediate coronary syndrome’

• 411.8 ‘Other acute coronary occlusion’

• 413.9 ‘Other and unspecified angina pectoris’ on or during hospitalization

2

Occurrence of at least one narrow diagnostic code ICD-9-CM:

• 410* ‘Acute myocardial infarction’a

3

Definition Acute Myocardial Infarction #2 AND Occurrence of at least one diagnostic procedure code within 30 days prior to diagnostic codea OR Occurrence of at least one therapeutic procedure code within 60 days after the diagnostic codec

4

American College of Cardiology/European Society of Cardiology Consensus Definition:

Occurrence of the following lab test results:

• Occurrence of the lab test ‘Blood Troponin’ LOINC codes 42757-5 or 10839-9, and lab test results ≥ the upper limit of normal in two sequential measurements, or ≥2 times the upper limit of normal in one measurement, and falling in a subsequent measurement. The ULN is defined as the 99th percentile of a non-MI control group, or 0.5 ng/mL if not available AND

• Occurrence of the lab test ‘Creatinine Phosphokinase MB Isozyme (CK-MB)’ LOINC codes 49551-5 or 13969-1, and lab test results ≥ the upper limit of normal in two sequential measurements, or ≥2 times the upper limit of normal in one measurement, and falling in a subsequent measurement. The ULN is defined as the 99th percentile of a non-MI control group, or 6 ng/mL if not available AND

Occurrence of an EKG test with the LOINC codes 11524-6, 8601-7, 18843-3, 18844-1, 18810-2, 8625-6 or 8634-8 within 10 days prior or after the lab test result and any of the following readings

• Any Q wave in leads V1 through V3, Q wave ≥5 to 30 ms in leads I, II, aVL, aVF, V4, V5, or V6 OR ST segment elevation: New or presumed new ST segment elevation at the J point in two or more contiguous leads with the cut-off of 0.2 mV in leads V1, V2, or V3 and ≥0.1 mV in other leads (contiguity in the frontal plane is defined by the lead sequence aVL, I, inverted aVR, II, aVF, III)

• ST segment depression OR T wave abnormalities

5

Definition Acute Myocardial Infarction #2 AND Hospitalization at date of diagnostic code

  1. aAn asterisk indicates a wildcard, i.e. any code with or without additional digits is included in the definition
  2. bA detailed list of all codes are available at http://omop.org/AcuteLiverInjury
  3. cA detailed list of all codes are available at http://omop.org/AcuteMyocardialInfarction

Rights and permissions

Reprints and permissions

About this article

Cite this article

Reich, C.G., Ryan, P.B. & Schuemie, M.J. Alternative Outcome Definitions and Their Effect on the Performance of Methods for Observational Outcome Studies. Drug Saf 36 (Suppl 1), 181–193 (2013). https://doi.org/10.1007/s40264-013-0111-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40264-013-0111-1

Keywords

Search

Navigation