Lalvani Shamal, Katsaggelos Aggelos
Department of Electrical and Computer Engineering, Northwestern University, Evanston, 60201, USA.
Sci Rep. 2024 May 17;14(1):11311. doi: 10.1038/s41598-024-61687-y.
Crowdsourcing involves the use of annotated labels with unknown reliability to estimate ground truth labels in datasets. A common task in crowdsourcing involves estimating reliabilities of annotators (such as through the sensitivities and specificities of annotators in the binary label setting). In the literature, beta or dirichlet distributions are typically imposed as priors on annotator reliability. In this study, we investigated the use of a neuroscientifically validated model of decision making, known as the drift-diffusion model, as a prior on the annotator labeling process. Two experiments were conducted on synthetically generated data with non-linear (sinusoidal) decision boundaries. Variational inference was used to predict ground truth labels and annotator related parameters. Our method performed similarly to a state-of-the-art technique (SVGPCR) in prediction of crowdsourced data labels and prediction through a crowdsourced-generated Gaussian process classifier. By relying on a neuroscientifically validated model of decision making to model annotator behavior, our technique opens the avenue of predicting neuroscientific biomarkers of annotators, expanding the scope of what may be learnt about annotators in crowdsourcing tasks.
众包涉及使用可靠性未知的注释标签来估计数据集中的真实标签。众包中的一个常见任务是估计注释者的可靠性（例如通过二元标签设置中注释者的敏感性和特异性）。在文献中，通常将贝塔分布或狄利克雷分布作为注释者可靠性的先验分布。在本研究中，我们研究了一种经神经科学验证的决策模型——漂移扩散模型，将其作为注释者标注过程的先验模型。我们对具有非线性（正弦）决策边界的合成生成数据进行了两个实验。变分推理用于预测真实标签和与注释者相关的参数。在众包数据标签预测以及通过众包生成的高斯过程分类器进行预测方面，我们的方法与一种先进技术（SVGPCR）表现相似。通过依靠经神经科学验证的决策模型来对注释者行为进行建模，我们的技术开辟了预测注释者神经生物标志物的途径，扩展了在众包任务中可以了解注释者的范围。
