International standards include recommendations that systematic reviews be comprehensive, but time and resources may render it impractical to search for and extract data from all possible sources of information. For example, many data sources exist for randomized controlled trials (RCTs), a number of which are not publicly available or are difficult or impossible to access. Searching nonpublic sources of RCTs may improve the impact of systematic reviews by identifying information that was recorded but not included in public sources, thus reducing the need for additional RCTs and improving research efficiency.
To determine whether multiple data sources about RCTs affected systematic reviews and meta-analyses of patient-centered outcomes (PCOs) research (eg, trial quality assessment, pooled effect estimates) and to produce open access guidance about using multiple data sources, for producers of systematic reviews of PCOs research.
We conducted a methods study related to the conduct of systematic reviews using 2 case studies: (1) gabapentin for neuropathic pain and (2) quetiapine for bipolar depression. Applying comprehensive searching methods as if we were conducting 2 systematic reviews, we attempted to identify all data sources for eligible RCTs. We extracted data and compared the information about trial design and trial quality (ie, risk of bias) using the multiple data sources. Using these multiple data sources, we extracted information about prespecified outcome domains, including the definition of each outcome (ie, the outcome domain, measure, metric, method of aggregation, and time point) and the associated results (ie, numerical estimates of treatment effectiveness). Then, we compared the results of meta-analyses using multiple data sources by conducting multiple meta-analyses in which we systematically added and replaced data from various sources. We compared the outcomes that matter most to patients with the outcomes that were examined in clinical trials.
Most clinical trials in our case studies were associated with multiple data sources, including public sources (eg, journal articles, conference abstracts, trial registrations, and Food and Drug Administration (FDA) reviews) and nonpublic sources (eg, clinical study reports and individual patient data). We found 21 gabapentin RCTs (74 reports, 6 individual participant data) and 7 quetiapine RCTs (50 reports, 1 individual participant data); we found nonpublic sources for 6 of 21 (29%) gabapentin and 4 of 7 (57%) quetiapine RCTs. We identified literally hundreds of PCOs reported in the sources we found that could be included in our meta-analyses. We surmised that there were many opportunities for selective outcome reporting by trialists and systematic reviewers. The process of identifying all sources of information—and extracting and analyzing data—required considerable time and skill. Nonpublic sources were especially difficult to identify. Clinical study reports were time consuming to extract, and individual participant data required time and skill to prepare the information for analysis. Data sources differed in completeness. Most RCTs (18/21 [86%] and 6/7 [86%], respectively) were reported in journal articles, which often presented unclear information related to trial quality. When nonpublic sources were available for RCTs, clinical study reports contained mostly information about trial design and trial quality, and clinical study reports and individual participant data contained the most results. In these case studies, individual participant data were not accompanied by any metadata (eg, codebooks or description of trial methods). For a single RCT reported in a single source, 1 outcome domain (eg, pain intensity) could be associated with multiple outcome definitions (eg, using multiple measures) and results. Thus, multiple results for an outcome could be presented selectively, even when an outcome domain was prespecified. Multiple data sources associated with a single RCT sometimes contained contradicting information about trial design. In the series of meta-analyses using results for a single outcome domain, measure, and time point for each case study, the effect of gabapentin on pain intensity ranged from an SMD of −0.46 (95% CI −0.64 to −0.28) to an SMD of −0.31 (95% CI −0.47 to −0.15), and the effect of quetiapine on depression ranged from SMD = −0.42 (95% CI −0.62 to −0.22) to SMD = −0.44 (95% CI −0.63 to −0.26). The range of results might have been wider had we included other outcome measures and time points in our meta-analyses. Clinical study reports and individual participant data contained hundreds of harms (adverse events) that were not included in public sources. Further analyses would be required to determine if conclusions about the relative benefits and harms of treatment would be affected by a systematic synthesis of harms included only in nonpublic sources.
There is tremendous variation in the information available across multiple data sources from individual trials. Reported estimates from individual trials were vulnerable to selective reporting at both the trial and the systematic review levels. Although an “open science” environment is a positive move forward, it will not solve all the problems that we identified with how trial information is currently made available. For example, an open trial data set for the purpose of making research reproducible allows for the investigation of new research questions, systematic reviews, and individual participant data meta-analysis; however, the steps required to share a data set and the steps required to utilize an open data set are resource intensive and demand skills that are not taught routinely to trialists or systematic reviewers, such as creating, storing, accessing, synthesizing, and analyzing information from multiple data sources. Furthermore, patients and stakeholders are often unfamiliar with the wide variety of existing data sources and what can be gained from each of them. Thus, current proposals to expand data sharing require careful and community-wide decision making about implementation. Although guidance for systematic reviewers suggests searching extensively for information sources and including all data sources found, this approach is neither practical nor possible for many systematic reviews. Our research provides practical insights into a complex area that eeds further research and discussion.
