BACKGROUND

Reports by PCORI and other publications on comparative effectiveness research (CER) highlight the need for observational data as a key tool for CER recommendations. Accomplishing this goal requires optimal application of statistical methods that account for nonrandomized treatment assignment and associated bias. Although numerous approaches exist for this purpose, as does substantial literature on their strengths and limitations, many fundamental questions remain for effectively applying these approaches in practice; 1 such question is, “Should I use propensity score-based methods, and if so, which variation of those methods should I use for my data set and research question?”

OBJECTIVES

The study had 3 main objectives: (1) systematically review existing literature and seek stakeholder input to identify studies from the literature on methods for causal inference in the setting of observational data used for CER; (2) analyze simulated data to further characterize statistical properties of common propensity score-based methods; and (3) develop and disseminate a Decision Tool for Causal Inference and Observational Data Analysis Methods in Comparative Effectiveness Research (DECODE CER) to guide clinical researchers and statisticians in using PS-based methods and instrumental variables (IVs).

METHODS

We conducted a systematic review to summarize the existing medical and statistical literature on methods for controlling bias and confounding when evaluating point interventions or treatments (ie, administered at a single time point) with observational data (where patients and/or physicians self-select treatment strategy). More specifically, the review sought to identify articles (from 1980-2014) from the medical and statistical literature that provided either simulation or theoretical results specific to statistical properties of causal inference methods in the setting of observational data. We then conducted additional simulations to further investigate statistical properties of PS-based approaches. Using the totality of available information gained from the articles identified in the systematic review, the simulations conducted for this study, and the published articles and textbooks on the subject, we created DECODE CER to provide guidance on using commonly implemented causal inference methods (ie, PS-based methods and IVs) for conducting CER with observational data.

RESULTS

The systematic review led to identifying 10 342 possible articles. After reviewing the abstracts, we reviewed the full articles of 772 publications to yield 168 articles that included simulations or theoretical results that evaluated properties of relevant methods. We conducted additional simulations to further compare PS methods with results from logistic regression models with variable selection. We created DECODE CER using Google Slides with embedded links to relevant resources specific to formulating a CER question, considering necessary design issues, assessing relevant assumptions, developing an analytical plan with common causal inference methods (eg, PS-based methods and IVs), and recognizing the role of other more complex methods. Investigators, an external advisory committee, clinical researchers, and trainees provided stakeholder input.

CONCLUSIONS

Without clear guidance on the connection between the research question, available data, and assumptions and properties of different causal inference methods, researchers often apply suboptimal methods, analytical findings suffer from serious flaws, and important topics in CER go unanswered or are answered incorrectly. Our literature search illustrated both the significant volume of available literature and the challenges that researchers face in trying to consider strengths and limitations of different methods for a given scenario. This tool does not seek to identify any “right answers”—rather, it organizes the available materials into a clear and easily accessible format on utilizing the most common causal inference approaches from the medical and statistical literature (PS-based methods and IVs). The main limitation of DECODE CER was the somewhat restricted scope of scenarios addressed (through the literature synthesis and simulations conducted); in future studies, we intend to expand these resources through collaboration with investigators of other ongoing PCORI methods projects.