Bailey David C
Department of Physics , University of Toronto , Toronto, Ontario, Canada M5S 1A7.
R Soc Open Sci. 2017 Jan 11;4(1):160600. doi: 10.1098/rsos.160600. eCollection 2017 Jan.
Judging the significance and reproducibility of quantitative research requires a good understanding of relevant uncertainties, but it is often unclear how well these have been evaluated and what they imply. Reported scientific uncertainties were studied by analysing 41 000 measurements of 3200 quantities from medicine, nuclear and particle physics, and interlaboratory comparisons ranging from chemistry to toxicology. Outliers are common, with 5 disagreements up to five orders of magnitude more frequent than naively expected. Uncertainty-normalized differences between multiple measurements of the same quantity are consistent with heavy-tailed Student's -distributions that are often almost Cauchy, far from a Gaussian Normal bell curve. Medical research uncertainties are generally as well evaluated as those in physics, but physics uncertainty improves more rapidly, making feasible simple significance criteria such as the 5 discovery convention in particle physics. Contributions to measurement uncertainty from mistakes and unknown problems are not completely unpredictable. Such errors appear to have power-law distributions consistent with how designed complex systems fail, and how unknown systematic errors are constrained by researchers. This better understanding may help improve analysis and meta-analysis of data, and help scientists and the public have more realistic expectations of what scientific results imply.
判断定量研究的意义和可重复性需要充分理解相关的不确定性，但这些不确定性的评估程度以及它们意味着什么往往并不明确。通过分析来自医学、核物理和粒子物理的3200个量的41000次测量，以及从化学到毒理学的实验室间比较，对报告的科学不确定性进行了研究。异常值很常见，出现5次分歧的频率比天真预期的高出五个数量级。同一量的多次测量之间的不确定性归一化差异与重尾学生分布一致，这种分布通常几乎是柯西分布，与高斯正态钟形曲线相差甚远。医学研究的不确定性通常与物理学中的不确定性评估得一样好，但物理学中的不确定性改善得更快，使得诸如粒子物理学中的5倍发现惯例等简单的显著性标准变得可行。错误和未知问题对测量不确定性的贡献并非完全不可预测。此类误差似乎具有幂律分布，这与设计的复杂系统如何失效以及研究人员如何约束未知的系统误差是一致的。这种更好的理解可能有助于改进数据的分析和荟萃分析，并帮助科学家和公众对科学结果的含义有更现实的期望。
