Veen Duco, Stoel Diederick, Schalken Naomi, Mulder Kees, Van de Schoot Rens
Department of Methods and Statistics, Utrecht University, 3584 CH 14 Utrecht, The Netherlands.
ProfitWise International, 1054 HV 237 Amsterdam, The Netherlands.
Entropy (Basel). 2018 Aug 9;20(8):592. doi: 10.3390/e20080592.
Experts' beliefs embody a present state of knowledge. It is desirable to take this knowledge into account when making decisions. However, ranking experts based on the merit of their beliefs is a difficult task. In this paper, we show how experts can be ranked based on their knowledge and their level of (un)certainty. By letting experts specify their knowledge in the form of a probability distribution, we can assess how accurately they can predict new data, and how appropriate their level of (un)certainty is. The expert's specified probability distribution can be seen as a prior in a Bayesian statistical setting. We evaluate these priors by extending an existing prior-data (dis)agreement measure, the Data Agreement Criterion, and compare this approach to using Bayes factors to assess prior specification. We compare experts with each other and the data to evaluate their appropriateness. Using this method, new research questions can be asked and answered, for instance: Which expert predicts the new data best? Is there agreement between my experts and the data? Which experts' representation is more valid or useful? Can we reach convergence between expert judgement and data? We provided an empirical example ranking (regional) directors of a large financial institution based on their predictions of turnover.
专家的信念体现了当前的知识状态。在做决策时考虑这些知识是很有必要的。然而，根据专家信念的价值对专家进行排名是一项艰巨的任务。在本文中，我们展示了如何根据专家的知识以及他们的（不）确定程度对专家进行排名。通过让专家以概率分布的形式指定他们的知识，我们可以评估他们预测新数据的准确程度，以及他们的（不）确定程度是否恰当。专家指定的概率分布在贝叶斯统计环境中可以被视为一种先验。我们通过扩展现有的先验 - 数据（不）一致度量标准——数据一致性准则来评估这些先验，并将这种方法与使用贝叶斯因子来评估先验规范进行比较。我们将专家相互之间以及与数据进行比较，以评估它们的恰当性。使用这种方法，可以提出并回答新的研究问题，例如：哪位专家对新数据的预测最佳？我的专家与数据之间是否一致？哪些专家的表述更有效或有用？我们能否在专家判断和数据之间达成一致？我们提供了一个基于大型金融机构（区域）董事对营业额的预测进行排名的实证例子。
