Investigating the accuracy of Bayesian techniques for before-after safety studies: the case of a "no treatment" evaluation.

The main challenge in conducting observational before-after (BA) studies of road safety measures is to use a methodology that accounts for many potential confounding factors. However, it is usually difficult to evaluate and decide on the accuracy of the different safety evaluation techniques available in literature. This is mainly because the outcome of the comparison has no specific target (i.e., the effect of a specific treatment on safety is not precisely known). The objective of this paper is to compare the accuracy of some of the commonly used Bayesian methodologies for BA safety studies by applying them to locations where no safety treatment has been implemented (making the target result to be no effect). This goal was pursued within the setting of a specific case study where a recent set of collision data was available for urban signalized intersections in British Colombia (Canada) with no safety treatments implemented during the time frame considered. An assessment of the temporal stability of the data set was undertaken to exclude the presence of significant BA changes as explanation of the results reported in this paper. Both the well-known empirical Bayes and the full Bayes method with non-linear intervention models were explored for comparison. Two types of selection of the hypothetical treatment sites were used in the analysis: random, to minimize the selection bias effect, and non-random, by selecting sites with abnormal collision frequency (hotspots). Several criteria were used for comparisons including variability among the index of effectiveness for individual treatment locations, the stability of the outcome in terms of the consistency of the results of several experiments and the overall treatment effectiveness. The results showed that when sites are selected randomly for treatment, all methodologies including the simple (naïve) BA study provide reasonable results (small statistically non-significant change in collision frequency). However, when sites are selected for treatment because of high collision occurrence, the estimated index of treatment effectiveness can potentially be biased by values up to 10%. This finding can have significant impact on estimating safety benefits of treatments, especially on those that have low collision reductions. As well, the FB method seems to perform better than other evaluation techniques including the most commonly used EB method. In particular, the FB method provides higher consistency in the estimated collision reduction among treatment sites.