Introduction, Tutorial, and Overview of Project
Bayesian statistics is an analytical approach that integrates prior knowledge and experience with current data to draw inferences about a specific quantity of interest This method relies on Bayes' theorem to update the probability of a hypothesis as more evidence becomes available.
Bayesian approaches offer a structured method for learning from accumulating evidence, but historically faced challenges in clinical trial design due to their computational intensity and controversial use of prior information Recent advancements in computational algorithms have largely addressed these limitations, making Bayesian methods more accessible As a result, the advantages of Bayesian techniques, particularly when robust prior information is available, have led to their growing popularity in the clinical trial community.
In 2006, the FDA's Center for Devices and Radiological Health (CDRH) issued draft guidance on the use of Bayesian statistics in medical device clinical trials, highlighting the growing acceptance of Bayesian methods in clinical research While this guidance offers a solid overview of Bayesian statistics and their application in trial design and analysis, it primarily addresses their use during the FDA approval process, neglecting the subsequent evaluations by the Centers for Medicare & Medicaid Services (CMS) regarding evidence for coverage decisions Furthermore, critics argue that the current FDA CDRH guidance overly emphasizes aligning Bayesian findings with traditional frequentist methods, which may limit the full potential of Bayesian approaches in clinical trials.
As Bayesian statistical methods become increasingly recognized in clinical trials, the Centers for Medicare & Medicaid Services (CMS) aims to evaluate their influence on policy decisions The CMS Coverage and Analysis Group has commissioned a report from The Technology Assessment Program (TAP) at the Agency for Healthcare Research and Quality (AHRQ) This report has been entrusted to the Duke Evidence-based Practice Center (EPC) under Contract Number: HHSA.
The primary objective of this project is to offer CMS a comprehensive framework for evaluating the application of Bayesian techniques in its evidence-based policy processes, supported by three specific aims to achieve this goal.
This report synthesizes existing research on the advantages and disadvantages of Bayesian techniques in clinical trial design and analysis, emphasizing their impact on policy-level decision-making A glossary of key terms, which are highlighted in bold throughout the report, is included to aid understanding.
This article investigates the application of Bayesian techniques within the context of Clinical Management Systems (CMS), focusing specifically on the prevention of sudden cardiac death (SCD) trials The aim is to assess the effective utilization of implantable cardioverter defibrillators (ICDs) in these clinical settings.
The goal is to leverage insights from previous investigations to draw specific lessons for the CMS context This includes evaluating the incorporation of Bayesian techniques in studies, identifying the situations where these methods are most applicable, and exploring how they can be effectively integrated with other data sources available to CMS, such as registries.
This report begins with an overview of its structure, followed by a fundamental tutorial on Bayesian statistical methods and their application in the design and analysis of clinical trials.
Bayesian analyses have been explored in various aspects of clinical trial design and analysis, including trial planning, execution, and subsequent trials They also facilitate the combination of data from multiple trials and the integration of registry data into the evidence base The literature review in Chapter 3 summarizes the diverse applications of Bayesian approaches and compares their advantages and disadvantages with traditional techniques.
This report primarily examines the application of Bayesian analysis in subgroup analysis within individual and multiple trials, particularly from the perspective of the Centers for Medicare & Medicaid Services (CMS) This focus is relevant as CMS frequently encounters subgroup analyses indicating varying effectiveness of drugs or devices for different patient categories, especially for those aged 65 and older Additionally, subgroup results often stem from small sample sizes or inconsistent data, necessitating the incorporation of supplementary information for reliable conclusions In Chapter 2, we outline four decision-making contexts where CMS might utilize Bayesian methods, and we consistently relate our findings to these contexts throughout the analysis.
After defining these contexts, we provide a review of the literature, describing current knowledge of subgroup analyses from both the Bayesian and frequentist perspectives
This article investigates the conditions under which Bayesian and frequentist statistical methods offer advantages in the design and analysis of Phase III efficacy trials We provide a summary of the existing literature that explores these methodologies, highlighting their respective strengths and applications in clinical research.
Bayesian techniques of clinical trial design and analysis could modify inferences and potentially affect CMS policy-level decisionmaking
This article explores the application of findings to clinical trials focused on implantable cardioverter defibrillator (ICD) therapy for preventing sudden cardiac death (SCD) By utilizing both simulation studies and a case study analyzing patient-level data from eight ICD trials, we examine the benefits and drawbacks of Bayesian techniques in contrast to frequentist methods These simulations serve to enhance and support the existing literature on this topic.
This article utilizes data from the ICD trials to demonstrate how analyses can be approached from both Bayesian and frequentist perspectives The primary objective is to help readers visualize the process and reporting of Bayesian analysis To showcase the types of data analysts may encounter, the case study includes analyses of both raw and summary data Additionally, we examine the application of Bayesian statistical techniques in clinical domains where registry data is available, particularly in areas where the Centers for Medicare & Medicaid Services (CMS) mandates the collection of supplementary patient data as a condition for national coverage decisions.
This report emphasizes that while the simulation studies and case study primarily examine clinical trials of Implantable Cardioverter Defibrillator (ICD) usage for the primary and secondary prevention of Sudden Cardiac Death (SCD), the findings are applicable across various clinical domains The report concludes with insights drawn from a comprehensive review of existing literature, simulation studies, and the case study.
Framing the Problem: CMS Contexts (or “Situations”)
We identified four decision-making contexts in which the CMS might utilize Bayesian approaches, and our analysis consistently relates our findings to these specific situations.
Applicants are seeking reimbursement from CMS for specific subgroups of patients, despite presenting results that indicate minimal or no overall efficacy of an intervention for the entire population.
• Situation 2: Applicants present CMS with results that suggest that an intervention is effective overall, but concern is raised that the benefits might be less effective in some subgroups
Applicants present CMS with findings indicating that an intervention is effective; however, the trial was conducted on a different demographic, specifically patients aged 55 to 64 They seek to apply these results to the patient population that CMS is focused on.
In situations where prior trials have shown effectiveness in high-risk populations, applicants are now proposing a new trial focused on a lower-risk population that is of interest to CMS They seek feedback on their proposed trial design and analysis to ensure the study's relevance and robustness.
For the purposes of this work, we assume that CMS’s evaluation task in each of the above situations involves three key steps:
1) Translating CMS’ general criterion of whether a given intervention is deemed
“reasonable and necessary” into specific criteria describing the outcomes that are necessary and sufficient to characterize the intervention’s value to the target population
2) Assessing the degree to which the intervention in question promotes improvements in those outcomes to the target populations
3) Judging whether those improvements are sufficient to implement into policy
Evaluation tasks can be conducted through two main approaches: frequentist statistical techniques and Bayesian techniques The first step in both methods involves establishing specific criteria for assessing an intervention The second step focuses on analyzing evidence, often utilizing frequentist statistical tools to determine levels of statistical significance The third step combines both quantitative and qualitative methods; quantitative methods may include criteria like the number of trials with a p-value below a certain threshold or more complex meta-analyses In contrast, qualitative methods aim to facilitate decision-making by gauging the "sense of the committee," which can be conducted informally or through structured approaches like the modified Delphi method.
What is distinctive about the two approaches is the way they address the latter two
The Bayesian approach integrates the assessment of evidence for decision-making, emphasizing the evaluation of its adequacy Specifically, a Bayesian analysis concentrates on estimating the effectiveness of the available evidence to inform actions.
“strength of belief” regarding any particular measure, for example, “Study X leads me to be Y percent confident that the effect of the intervention is greater than Z”
The Bayesian approach facilitates a coherent interpretation of multiple studies, contributing to a comprehensive body of evidence and linking various forms of data for aggregate inferences Ultimately, the focus for CMS and society is on enhancing health outcomes for Medicare beneficiaries Evaluations are conducted within a complex landscape involving diverse stakeholders with varying interests, making it essential that evaluation strategies align with existing decision-making contexts Beyond merely deriving accurate inferences, an effective evaluation strategy should also prioritize transparency, clarity, efficiency, and the accommodation of multiple objectives.
Literature Review
This report examines the application of Bayesian techniques in the context of CMS policymaking, specifically investigating whether Bayesian or frequentist statistical methods offer advantages in the design or analysis of Phase III efficacy trials The literature review was conducted to identify circumstances where these approaches could influence inferences that impact policy-level decisions While our simulation studies and a case study in the ICD clinical domain also address this issue, we prioritized reviewing existing published literature for empirical evidence.
We conducted a comprehensive search of MEDLINE® using keywords associated with Bayesian theory, frequentist analysis, and health policy, focusing exclusively on English-language trials and review articles Additionally, we explored the reference lists of significant papers and proceedings from a recent SAMSI workshop on subgroup analysis to identify further relevant publications The titles and abstracts of all identified studies were independently reviewed by two investigators to ensure thorough evaluation.
The following types of articles were excluded:
• Epidemiological studies (observational or longitudinal studies)
• Randomized controlled trials (RCT) that did not include Bayesian analysis
Meta-analyses and cost-effectiveness analyses were selected for review based on their focus on methods that facilitate a comparison between Bayesian and frequentist approaches During the title-and-abstract screening phase, any article was advanced to full-text review if at least one of the two reviewers recommended its inclusion.
During the full-text review stage, two independent reviewers reassessed the articles, including those that aligned with one or more of the specified topics of interest Any disagreements between the reviewers were addressed through discussion to reach a consensus.
Our comprehensive search strategies yielded 334 potentially relevant citations After the title-and-abstract screening, 197 citations were excluded, followed by an additional 67 exclusions during the full-text screening Ultimately, 70 studies were included for review.
The literature review identified four key themes regarding Bayesian techniques: first, it examined the advantages and disadvantages of these methods in clinical trial design and analysis; second, it explored their application in subgroup analyses; third, it highlighted the use of Bayesian techniques in meta-analysis; and finally, it assessed the impact of employing Bayesian methods on research outcomes.
Table 1 reports the number of included articles reviewed for each of the four themes Note that some articles were included for more than one theme
In what follows, we summarize our review of the literature in these four themes – while focusing these summaries on areas of interest to CMS
Advantages and Disadvantages of Bayesian Techniques in Clinical Trial Design and Analysis
Potential Advantages of Bayesian Approaches
The statistical literature offers a wealth of resources on Bayesian theory and its application in medicine, highlighting both the benefits and drawbacks of Bayesian techniques in clinical trial design and analysis This discussion does not aim to cover all aspects or serve as a complete introduction to Bayesian statistics For an in-depth overview of Bayesian approaches in clinical trials, readers are encouraged to consult the comprehensive summary by Spiegelhalter and colleagues, as well as the Health Technology Assessment from the National Institute for Health Research.
A comprehensive review of Bayesian methods in health technology assessment can be found in the NHS guidelines and the 2006 FDA guidance for medical device trials The advantages and disadvantages of these Bayesian approaches are primarily derived from these key sources Additionally, the International Society for Bayesian Analysis (ISBA) offers a valuable compilation of Bayesian resources for further exploration.
The CMS decision-making process primarily addresses scenarios where clinical trials have been conducted, evaluating the sufficiency of the existing evidence base Bayesian methods can enhance decision-making in two key areas: subgroup analysis and the meta-analysis of accumulating clinical evidence Additionally, Bayesian approaches offer three notable advantages: the incorporation of prior information, determination of sample sizes, and the use of adaptive designs It is crucial to recognize that, like frequentist methods, Bayesian clinical trials require rigorous planning and analysis By leveraging Bayes’ theorem, Bayesian statistics enable the integration of prior knowledge with current data, resulting in an updated understanding of the quantity of interest While the use of prior information is a significant benefit of Bayesian techniques, it also raises concerns among frequentist trialists.
Bayesian methods can be contentious when prior information relies heavily on personal opinions or subjective expert judgments In these cases, conducting sensitivity analyses on prior distributions becomes crucial Conversely, using prior information grounded in empirical evidence from existing clinical trials is generally accepted, particularly in the context of CMS To ensure appropriateness, the evidential basis of the prior, along with any potential biases, must be clearly outlined Furthermore, many experts stress the importance of sensitivity analyses to examine various options for the selected prior.
Fisher discusses the differences between Bayesian and frequentist analysis in clinical trials, highlighting controversies surrounding the use of prior information and the challenges in its elicitation and integration into existing evidence Notable studies that examine the influence of prior information on clinical trials include research by Gennari et al., Tyson et al., Brophy and Joseph, and Kpozehouen et al.
Bayesian statistical approaches offer significant advantages in trial design and analysis, particularly through the incorporation of informative priors Even in the absence of such priors, Bayesian methods remain valuable by allowing for interim analyses and midcourse adjustments These approaches can influence the sample size needed for demonstrating sufficient evidence to the Centers for Medicare & Medicaid Services (CMS) Adjustments may stem from prior information or interim evaluations during the clinical trial As highlighted by Schmid and colleagues, utilizing prior evidence can potentially decrease the required sample size if it provides insights into the effect size.
When prior evidence indicates uncertainty about effect size, increasing the sample size may be necessary Both Bayesian and frequentist statistical methods aim to collect sufficient information to evaluate an intervention's efficacy while minimizing resource waste and patient risk Bayesian methods do not require a fixed sample size; instead, they establish criteria for trial cessation Throughout the trial, Bayesian techniques can assess the posterior distribution for sample size and determine the additional observations needed to meet stopping criteria, allowing for the trial to conclude when adequate information is available to address the clinical or policy question An example of Bayesian sample size determination is illustrated by Wang and colleagues.
Bayesian approaches enable the incorporation of adaptive designs in clinical trials, allowing for the midcourse elimination of unfavorable treatment arms and modifications to randomization schemes While frequentist methods include sequential analysis techniques, the Bayesian framework is particularly advantageous for interim reviews The decision to halt a trial based on interim analysis should consider the costs and benefits of enrolling additional subjects Lewis and colleagues highlight the challenges of making such comparisons with frequentist methods, providing examples of Bayesian applications In Bayesian monitoring, a trial's outcome can be deemed positive or negative if the posterior probability of a significant improvement exceeds a predetermined threshold, relying on interim and predicted future data Dmitrienko and Wang, along with Freedman and Spiegelhalter, explore Bayesian strategies for monitoring clinical trials, emphasizing the importance of prior distribution choices Their findings indicate that weak priors can lead to early stopping in futility monitoring, which is beneficial in large mortality trials to protect critically ill patients from ineffective treatments However, using weak priors in proof-of-concept studies may result in excessively high early termination rates.
In these situations, stronger aggressive (i.e informative) priors are preferable 14
Emerson and colleagues emphasize the significance of incorporating diverse prior distributions in Bayesian stopping rules Dignam and colleagues discuss a contentious trial stoppage based on interim results, illustrating how a Bayesian approach facilitates the exploration of various prior beliefs about treatment efficacy and the justification for early trial termination Additionally, George and colleagues, along with Berry and colleagues, present further examples of employing Bayesian statistical methods to halt clinical trials early, highlighting the distinctions between this approach and frequentist techniques.