An Attempt to Replicate Randomized Trials of Diabetes Treatments Using a Japanese Administrative Claims and Health Checkup Database: A Feasibility Study.

BACKGROUND

Use of real-world evidence (RWE) has been limited for evaluating effectiveness because of the lack of confidence in its reliability. Examining whether a rigorously designed observational study using real-world data (RWD) can reproduce the results of a randomized controlled trial (RCT) will provide insights into the implementation of high-quality RWE studies that can produce valid conclusions.

OBJECTIVE

We aimed to replicate published RCTs using a Japanese claims and health checkup database and examine whether the emulated RWE studies' results agree with those of the original RCTs.

METHODS

We selected three RCTs on diabetes medications for replication in patients with type 2 diabetes. The study outcome was either the change or percentage change in HbA1c levels from baseline. We designed three observational studies using the RWD to mimic the critical study elements of the respective RCTs as closely as possible. We performed 1:1 propensity score nearest-neighbor matching to balance the groups for potential confounders. The differences in outcomes between the groups and their 95% confidence intervals (CIs) were calculated in each RWE study, and the results were compared with those of the RCT.

RESULTS

Patient characteristics, such as age, sex, and duration of diabetes, differed between the RWE studies and RCTs. In Trial 1 emulation, the percentage changes in HbA1c levels were larger in the treatment group than in the comparator group (difference -6.21, 95% confidence interval (CI) -11.01 to -1.40). In Trial 2, the change in HbA1c level was larger in the treatment group (difference -0.01; 95% CI -0.25 to 0.23), and in Trial 3, it was smaller in the treatment group (difference 0.46; 95% CI -0.01 to 0.94). These results did not show regulatory or estimate agreement with the RCTs.

CONCLUSIONS

None of the three emulated RWE studies using this claims and health checkup database reproduced the same conclusions as the RCTs. These discrepancies could largely be attributed to design differences between RWE studies and RCTs, primarily due to the lack of necessary data in the database. This particular RWD source may not be the best fit for evaluating treatment effects using laboratory data as the study outcome.