Advertisement

Conducting Comparative Effectiveness Research on Medications: The Views of a Practicing Epidemiologist from the Other Washington

  • Bruce M. Psaty
    Correspondence
    Address correspondence to: Bruce M. Psaty, Cardiovascular Health Research Unit, 1730 Minor Avenue, Suite 1360, Seattle, WA 98177, USA
    Affiliations
    Cardiovascular Health Research Unit, Departments of Medicine, Epidemiology, and Health Services, University of Washington, Seattle, WA, USA; Group Health Research Institute, Group Health Cooperative, Seattle, WA, USA
    Search for articles by this author
Open ArchivePublished:November 10, 2011DOI:https://doi.org/10.1016/j.jval.2011.08.1740

      Keywords

      Comparative-effectiveness research (CER), which is not a new concept, has succeeded evidence-based medicine as the current fashion. In the old days of evidence-based medicine, active treatment comparators were not banned and one of the major concerns was the choice of outcomes important to and for patients. For instance, high blood pressure itself is an asymptomatic condition, but its devastating clinical complications such as a heart attack or a stroke are not. So, the two movements are similar, one succeeding the other. Perhaps the major difference is the existence of the legislative mandate (PL 111-148), which at once enables and constrains CER.
      In a Medline search, I found 44,344 citations for the venerable discipline of evidence-based medicine, 17,140 for the up-and-coming genre of CER, but only 8 for the much-avoided option of opinion-based medicine. No one I know boasts a serious allegiance to opinion-based medicine. This one-sided distribution of published studies strikes epidemiologists like me as odd. Evidence is generally clear; but its validity, interpretation, and generalizability remain a source of active contention. The appeals to evidence, reason, and science are almost universal; but these rhetorical devices may do little to clarify the nature of the process and the disagreements, which often reflect differences in values and assumptions. Indeed, the models of the behavioral economists may provide more insight into this historical process than the rational-choice models of the traditional economists.
      For example, the Systolic Hypertension Evaluation Program (SHEP) randomized older adults to low-dose diuretics or placebo, and the results were published in 1991 [
      SHEP Cooperative Research Group
      Prevention of stroke by antihypertensive drug treatment in older persons with isolated systolic hypertension: final results of the Systolic Hypertension in the Elderly Program (SHEP).
      ]. Low-dose diuretics were associated with a major reduction in the risk of stroke or heart attack. The number needed to treat to prevent one serious cardiovascular event was about 120 persons for 1 year. Insofar as evidence matters, if we properly apply the findings from a treatment trial of low-dose diuretics, the expected effect of SHEP results on the hypertension drug treatment patterns of older adults in the United States is a dramatic increase in the use of diuretics in this patient group.
      The actual trends in antihypertensive drug use after the publication of SHEP results raise questions about the relationship between evidence and practice, one that may limit the impact of evidence generated from CER studies. In a report from the Cardiovascular Health Study, a cohort study of older adults [
      • Psaty B.M.
      • Manolio T.A.
      • Smith N.L.
      • et al.
      Time trends in high blood pressure control and the use of antihypertensive medications in older adults: the Cardiovascular Health Study.
      ], the use of thiazide diuretics dropped precipitously from 60% in 1990 to 38% in 1999. This trend looks like the public-health response to a major safety alert and not an appropriate response to new evidence of the efficacy of low-dose diuretics. Before late 1997, there was no evidence of health benefits associated with the use of calcium antagonists and angiotensin-converting enzyme (ACE) inhibitors [
      • Staessen J.A.
      • Fagard R.
      • Thijs L.
      • et al.
      Systolic Hypertension in Europe Trial Investigators
      Randomised double-blind comparison of placebo and active treatment for older patients with isolated systolic hypertension.
      ]; yet despite the absence of evidence of efficacy and safety, the use of both drug classes increased from about 15% in 1990 up to about 37% in 1999. SHEP evaluated low-dose diuretics; but without any evidence, the findings were generalized to all forms of drug treatment for hypertension. These trends reflect not the play of evidence but the power of marketing. Despite the safety, the effectiveness, and the low cost of diuretics—a month's supply costs less than a cup of Starbuck's coffee—the rational-choice theory does not serve as a good model of the medical community's response to the emergence of high-quality evidence from SHEP, a high-quality randomized trial.
      As the number of new therapies increases, comparative studies become especially important. My first epidemiologic study was an observational case–control study of the association between antihypertensive drug use and the risk of coronary heart disease [
      • Psaty B.M.
      • Koepsell T.D.
      • LoGerfo J.P.
      • et al.
      Beta-blockers and primary prevention of coronary heart disease in patients with high blood pressure.
      ,
      • Psaty B.M.
      • Koepsell T.D.
      • Wagner E.H.
      • et al.
      The relative risk of incident coronary heart disease associated with recently stopping the use of beta-blockers.
      ]. At the time, randomized trials that had evaluated high-dose diuretics against placebo in patients with hypertension showed reduced risks of major cardiovascular events [
      Hypertension Detection and Follow-up Program Cooperative Group
      Five-year findings of the Hypertension Detection and Follow-up Program, I: reduction in mortality of persons with high blood pressure, including mild hypertension.
      ]. In postmyocardial infarction patients, the randomized trials had provided evidence of the health benefits associated with the use of beta-blockers [
      Beta-blocker Heart Attack Trial Research Group
      A randomized trial of propranolol in patients with acute myocardial infarction, I: mortality results.
      ]. Although both diuretics and beta-blockers were approved for the treatment of hypertension, all the active-treatment comparative trials, save one [
      Medical Research Council Working Party
      Medical Research Council trial of treatment of hypertension in older adults: principal results.
      ], had focused only on the short-term outcomes of levels of blood pressure or side effects. As a young epidemiologist, I was struck by the extraordinary research opportunities provided by the electronic administrative records of an HMO such as Group Health. Specifically, they included the possibility of 1) complete case ascertainment, 2) population-based controls, and 3) computerized pharmacy records to assess drug use. With the advent of electronic medical records, some of these same features are attractive to CER scientists.
      These pharmacoepidemiological studies are early examples of observational CER. The primary rationale for the use of a comparative design was an epidemiological one. We wanted to avoid confounding by indication [
      • Psaty B.M.
      • Koepsell T.D.
      • Lin D.Y.
      • et al.
      Assessment and control for confounding by indication in observational studies.
      ]. In an observational study of hypertension treatment, it is simply not possible to compare treated and untreated patients for cardiovascular outcomes. The treated patients have hypertension that is more severe, more difficult to control, and more resistant to nonpharmacologic measures. In the absence of randomization, a naive observational comparison between treated and untreated patients will paradoxically show an increased risk of cardiovascular events associated with antihypertensive drug treatment. The use of an active-treatment comparison group reduces but does not eliminate the problem of confounding by indication, and so other techniques, including restriction and adjustment, are also necessary.
      We were interested in both the benefits and the harms, and the comparative element was useful even in the evaluation of safety, where confounding by indication is less likely to be a problem. Stopping the use of beta-blockers, tracked by the pharmacy data, was associated with a transient increase in the risk of coronary heart disease [
      • Psaty B.M.
      • Koepsell T.D.
      • Wagner E.H.
      • et al.
      The relative risk of incident coronary heart disease associated with recently stopping the use of beta-blockers.
      ]. In the safety study, the increased risk was specific to stopping beta-blockers but not to stopping diuretics.
      In the evaluation of medications, even though randomized trials are the preferred design, observational studies may on occasion have a role to play. Partly because of the results of SHEP and the marketing efforts of the industry, the 1990s witnessed an epidemic of licit drug use for the treatment of hypertension. The Food and Drug Administration (FDA) had approved antihypertensive drugs solely on the basis of their ability to lower blood pressure in short-term studies. For new classes of medications, placebo-controlled trials evaluating major health outcomes had not been conducted. Given the different biological mechanisms for the four major drug classes—diuretics, beta-blockers, ACE inhibitors, and calcium antagonists—it would be astonishing if they were all to have the same effect on outcomes such as myocardial infarction, heart failure, and stroke. Manufacturers were aggressively advertising the new ACE inhibitors and calcium antagonists, often on the basis of side-effect profiles. Indeed, marketing had driven the prevalence of use so high that another observational study of the comparative effectiveness of major health outcomes was possible.
      In our case–control study [
      • Psaty B.M.
      • Heckbert S.R.
      • Koepsell T.D.
      • et al.
      The risk of myocardial infarction associated with anti-hypertensive drug therapies.
      ], short-acting calcium antagonists were associated with an increased risk of myocardial infarction. The association was robust. We could not make it go away. In the meta-analysis of randomized trials [
      • Furberg C.D.
      • Psaty B.M.
      • Myers J.V.
      Nifedipine: dose-related increase in mortality in patients with coronary heart disease.
      ], the short-acting nifedipine was associated with a dose-related increase in the risk of total mortality. Well aware of the limitations of observational studies, we used the results to advocate for the conduct of large high-quality, long-term randomized trials to evaluate the comparative health risks and benefits of drugs that were being used by tens of millions of people. Although or perhaps because we conduct observational studies, we believed then and still believe that the optimal method of evaluating drugs for effectiveness is the randomized trial.
      The response to our CER epidemiological studies may have been shaped by other cultural influences of the time. These studies were published during a previous era of partisan bitterness when Newt Gingrich had came to power in the House and legitimized an aggressive take-no-prisoners style of discourse [
      • Deyo R.A.
      • Psaty B.M.
      • Simon G.
      • et al.
      The messenger under attack--intimidation of researchers by special-interest groups.
      ]. Manufacturers with strong financial interests issued, often by fax, critiques that attempted to discredit us and our studies. One pharmaceutical company used the Washington State freedom-of-information law to harass those of us who were state employees. Academic cardiologists, not usually known for their epidemiological wisdom, were paid large fees to wander about the country and opine solemnly about the design and conduct of case–control studies. Too often, the style resembled not scientific discourse but the aggressive marketing chatter, the partisan wrangling, and the combative attacks of political campaigns [
      • Deyo R.A.
      • Psaty B.M.
      • Simon G.
      • et al.
      The messenger under attack--intimidation of researchers by special-interest groups.
      ].
      Early in my career, I had received a grant award from the Society of Epidemiological Research, one that was funded by the Merck Company Foundation. At the time, Merck made ACE inhibitors but not calcium antagonists, and I was accused by the manufacturers of calcium antagonists of working for the ACE-inhibitor companies. These kinds of bizarre experiences persuaded me that if I wanted to work in the area of drug safety, I could not work safely for any drug manufacturer. For the most part, I have remained clean and sober since that time. As academic scientists, my colleagues and I nonetheless chose to capitalize on these experiences by writing about them in the New England Journal of Medicine [
      • Deyo R.A.
      • Psaty B.M.
      • Simon G.
      • et al.
      The messenger under attack--intimidation of researchers by special-interest groups.
      ].
      Perhaps most interesting are the unintended consequences. The pharmaceutical manufacturers effectively promoted and publicized what would otherwise have been several minor epidemiologic studies. For me personally, the decision not to seek or accept consulting from pharmaceutical companies provided novel future opportunities: The absence of financial conflicts of interest helped me to qualify as an expert witness at Senator Grassley's hearing on Vioxx (Merck & Co., Inc., Whitehouse Station, NJ) and for two Institute of Medicine committees on drug safety and the FDA. Unexpectedly and unintentionally, our work also turned out to be quite valuable for the industry as well. The attention to the risks associated with the generic short-acting calcium antagonists helped promote the switch to long-acting still-on-patent trade calcium antagonists. This narrative illustrates some of the ways in which the practice and history of medicine are not rational, scientific, or evidence-based activities. What I have described is a social and political process with various competing interests, some more powerful than others. Insights from the behavioral economists may provide guidance for shaping the future narrative.
      For the CER initiative, the quality of the data and the quality of the design may on occasion be more important than the traditional distinction between observational studies and randomized trials. In a commentary, Hennekens and Demets [
      • Hennekens C.
      • Demets D.
      The need for large-scale randomized evidence without undue emphasis on small trials, meta-analyses, or subgroup analyses.
      ] emphasize the need for large-scale randomized evidence. As a society, we do not want drugs widely used on the basis of questionable evidence. High-quality trials, including comparative trials, are essential to progress in American medicine. The authors have selected, however, an unfortunate example—the RECORD (Rosiglitazone Evaluated for Cardiac Outcomes and Regulation of Glycemia in Diabetes) trial [
      • Home P.D.
      • Pocock S.J.
      • Beck-Nielsen H.
      • et al.
      Rosiglitazone Evaluated for Cardiac Outcomes and Regulation of Glycemia in Diabetes (RECORD) Study: interim findings on cardiovascular hospitalizations and deaths.
      ]. This study compared the diabetes medication rosiglitazone with other active treatments in an open-label randomized trial. Even in the presence of a “blinded” adjudication panel, open-label trials create opportunities for bias that may be more extreme than the biases encountered in observational studies. As part of the review of rosiglitazone, Marciniak [
      • Marciniak T.A.
      Memorandum of June 14, 2010 on cardiovascular events in RECORD, NDA 21-071/S-035.
      ], an FDA scientist, reviewed the investigators' decisions to refer a sample of potential events to the blinded adjudication panel. He detected a total of 70 errors: 57 favored rosiglitazone and only 13 favored the control. In other words, the errors in outcome assessments were four times more likely to occur in favor of rosiglitazone than the other active treatments. Indeed, for both observational studies and randomized trials, the quality of the design often influences the quality of the data.
      As Rubin [
      • Rubin D.
      On the limitations of comparative effectiveness research.
      ] has emphasized, the design and conduct of observational CER is exquisitely difficult. Propensity scores, developed by Rosenbaum and Rubin [
      • Rosenbaum P.R.
      • Rubin D.B.
      The central role of the propensity score in observational studies for causal effects.
      ], will not transform observational studies into the evidence equivalent of randomized trials. A key feature of all studies is internal validity, without which the question of generalizability does not arise. To ensure internal validity in observational studies, restriction is often necessary and may severely limit the populations that can be included and evaluated confidently [
      • Psaty B.M.
      • Siscovick D.S.
      Minimizing bias due to confounding by indication in comparative effectiveness research: the importance of restriction.
      ]. In addition, confounding by indication is a major problem in the observational study of efficacy or effectiveness. For example, the reported reduction in mortality risk associated with influenza vaccine in an observational study is simply too large to be credibly explained by the attack rate and the case-fatality rate of influenza [
      • Nichol K.
      • Nordin J.
      • Nelson D.
      • et al.
      Effectiveness of influenza vaccine in the community-dwelling elderly.
      ,
      • Jackson L.A.
      • Nelson J.C.
      • Benson P.
      • et al.
      Functional status is a confounder of the association of influenza vaccine and risk of all cause mortality in seniors.
      ]. On the other hand, for safety outcomes, meta-analyses of observational studies and randomized trials have provided remarkably similar findings [
      • Golder S.
      • Loke Y.K.
      • Bland M.
      Meta-analyses of adverse effects data derived from randomised controlled trials as compared to observational studies: methodological overview.
      ]. Finally, data and design quality may be more important than the sample size. With the advent of large databases, we have the opportunity to conduct large simple observational studies, but in the absence of high-quality data, we may find ourselves with excessively precise and extremely biased estimates of associations.
      How will the CER initiative and the evidence that it produces influence the US health-care system? Rose [
      • Rose G.
      Sick individuals and sick populations.
      ] described the prevention paradox. Population-based interventions such as a reduction in salt intake in the diet can result in pronounced reductions in the incidence of disease in the population as a whole; but the benefits to each individual who participates are small. The treatment paradox occurs in the setting of high-risk individuals. Identifying them and treating them aggressively are effective; but this costly and increasingly high-tech approach has little impact on the incidence of disease in the population. American medicine is best at this “rescue” style of medical practice. The increase in health-care costs in America, driven by a large variety of vested interests ranging from diagnostics and therapeutics to hospitals and physicians, is unfortunately not caused by the absence of evidence from comparative-effectiveness studies. Although new comparative-effectiveness information may improve care or reduce costs in some settings, the incentives for implementation are often small or absent. In evidence development, it is important to use the best methods and recognize the limitations of the data. The results of observational studies can on occasion be used appropriately to advocate for randomized trials of effectiveness. The methods of advocacy, counterdetailing, and other population-based approaches will be essential for large-scale implementation.
      Fortunately, CER comprises all study designs. Notwithstanding the RECORD trial [
      • Home P.D.
      • Pocock S.J.
      • Beck-Nielsen H.
      • et al.
      Rosiglitazone Evaluated for Cardiac Outcomes and Regulation of Glycemia in Diabetes (RECORD) Study: interim findings on cardiovascular hospitalizations and deaths.
      ], randomized trials are the preferred method for evaluating the effectiveness of drug therapies. Those of us who hope that our research findings will affect policy have a duty to provide a reliable and valid answer. To believe without reservation that observational studies are often adequate to the task of evaluating the effectiveness of drugs may only serve paradoxically to jeopardize the health of the public. Observational studies are better suited to evaluate safety than effectiveness. Those of us who hope that our research findings will influence policy may also reasonably expect resistance or adversarial responses from those who have a vested interest in the products affected by the new evidence. Until controls on the perpetual use of new incompletely evaluated diagnostic and therapeutic strategies can be implemented, low expectations aimed at incremental progress in both the development and the implementation of evidence are likely to be key to ongoing success.

      Acknowledgments

      An earlier version of this article was delivered as an invited plenary session talk at the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) 16th Annual International Meeting in Baltimore, MD, May 2011.
      Source of financial support: This research was supported in part by grants HL078888, HL080295, HL085251, HL087652, and HL105756 from the National Heart, Lung, and Blood Institute. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Heart, Lung, and Blood Institute or the National Institutes of Health. Dr. Psaty serves on a data safety monitoring committee for a clinical trial funded by Zoll.

      References

        • SHEP Cooperative Research Group
        Prevention of stroke by antihypertensive drug treatment in older persons with isolated systolic hypertension: final results of the Systolic Hypertension in the Elderly Program (SHEP).
        JAMA. 1991; 265: 3255-3264
        • Psaty B.M.
        • Manolio T.A.
        • Smith N.L.
        • et al.
        Time trends in high blood pressure control and the use of antihypertensive medications in older adults: the Cardiovascular Health Study.
        Arch Intern Med. 2002; 162: 2325-2332
        • Staessen J.A.
        • Fagard R.
        • Thijs L.
        • et al.
        • Systolic Hypertension in Europe Trial Investigators
        Randomised double-blind comparison of placebo and active treatment for older patients with isolated systolic hypertension.
        Lancet. 1997; 350: 757-764
        • Psaty B.M.
        • Koepsell T.D.
        • LoGerfo J.P.
        • et al.
        Beta-blockers and primary prevention of coronary heart disease in patients with high blood pressure.
        JAMA. 1989; 261: 2087-2094
        • Psaty B.M.
        • Koepsell T.D.
        • Wagner E.H.
        • et al.
        The relative risk of incident coronary heart disease associated with recently stopping the use of beta-blockers.
        JAMA. 1990; 263: 1653-1657
        • Hypertension Detection and Follow-up Program Cooperative Group
        Five-year findings of the Hypertension Detection and Follow-up Program, I: reduction in mortality of persons with high blood pressure, including mild hypertension.
        JAMA. 1979; 242: 2562-2571
        • Beta-blocker Heart Attack Trial Research Group
        A randomized trial of propranolol in patients with acute myocardial infarction, I: mortality results.
        JAMA. 1982; 247: 1707-1714
        • Medical Research Council Working Party
        Medical Research Council trial of treatment of hypertension in older adults: principal results.
        BMJ. 1992; 304: 405-412
        • Psaty B.M.
        • Koepsell T.D.
        • Lin D.Y.
        • et al.
        Assessment and control for confounding by indication in observational studies.
        J Am Geriatr Soc. 1999; 47: 749-754
        • Psaty B.M.
        • Heckbert S.R.
        • Koepsell T.D.
        • et al.
        The risk of myocardial infarction associated with anti-hypertensive drug therapies.
        JAMA. 1995; 274: 620-625
        • Furberg C.D.
        • Psaty B.M.
        • Myers J.V.
        Nifedipine: dose-related increase in mortality in patients with coronary heart disease.
        Circulation. 1995; 92 ([published correction appears in Circulation 1996;93:1475–6]): 1326-1331
        • Deyo R.A.
        • Psaty B.M.
        • Simon G.
        • et al.
        The messenger under attack--intimidation of researchers by special-interest groups.
        N Engl J Med. 1997; 336: 1176-1180
        • Hennekens C.
        • Demets D.
        The need for large-scale randomized evidence without undue emphasis on small trials, meta-analyses, or subgroup analyses.
        JAMA. 2009; 302: 2361-2362
        • Home P.D.
        • Pocock S.J.
        • Beck-Nielsen H.
        • et al.
        Rosiglitazone Evaluated for Cardiac Outcomes and Regulation of Glycemia in Diabetes (RECORD) Study: interim findings on cardiovascular hospitalizations and deaths.
        N Engl J Med. 2007; 357: 28-38
        • Marciniak T.A.
        Memorandum of June 14, 2010 on cardiovascular events in RECORD, NDA 21-071/S-035.
        (FDA briefing document, pp 16–151) ([Accessed July 9, 2010])
        • Rubin D.
        On the limitations of comparative effectiveness research.
        Stat Med. 2010; 29 (discussion 1996–7): 1991-1995
        • Rosenbaum P.R.
        • Rubin D.B.
        The central role of the propensity score in observational studies for causal effects.
        Biometrika. 1983; 70: 41-55
        • Psaty B.M.
        • Siscovick D.S.
        Minimizing bias due to confounding by indication in comparative effectiveness research: the importance of restriction.
        JAMA. 2010; 304: 897-898
        • Nichol K.
        • Nordin J.
        • Nelson D.
        • et al.
        Effectiveness of influenza vaccine in the community-dwelling elderly.
        N Engl J Med. 2007; 357: 1373-1381
        • Jackson L.A.
        • Nelson J.C.
        • Benson P.
        • et al.
        Functional status is a confounder of the association of influenza vaccine and risk of all cause mortality in seniors.
        Int J Epidemiol. 2006; 35: 345-352
        • Golder S.
        • Loke Y.K.
        • Bland M.
        Meta-analyses of adverse effects data derived from randomised controlled trials as compared to observational studies: methodological overview.
        PLoS Med. 2011; 8: e1001026
        • Rose G.
        Sick individuals and sick populations.
        Int J Epidemiol. 1985; 14: 32-38