IEEE Trans Pattern Anal Mach Intell. 2019 Sep;41(9):2280-2286. doi: 10.1109/TPAMI.2018.2849989. Epub 2018 Jun 25.
By providing substantial amounts of data and standardized evaluation protocols, datasets in computer vision have helped fuel advances across all areas of visual recognition. But even in light of breakthrough results on recent benchmarks, it is still fair to ask if our recognition algorithms are doing as well as we think they are. The vision sciences at large make use of a very different evaluation regime known as Visual Psychophysics to study visual perception. Psychophysics is the quantitative examination of the relationships between controlled stimuli and the behavioral responses they elicit in experimental test subjects. Instead of using summary statistics to gauge performance, psychophysics directs us to construct item-response curves made up of individual stimulus responses to find perceptual thresholds, thus allowing one to identify the exact point at which a subject can no longer reliably recognize the stimulus class. In this article, we introduce a comprehensive evaluation framework for visual recognition models that is underpinned by this methodology. Over millions of procedurally rendered 3D scenes and 2D images, we compare the performance of well-known convolutional neural networks. Our results bring into question recent claims of human-like performance, and provide a path forward for correcting newly surfaced algorithmic deficiencies.
通过提供大量的数据和标准化的评估协议，计算机视觉中的数据集帮助推动了视觉识别各个领域的进展。但即使考虑到最近基准测试中的突破结果，我们仍然有理由怀疑我们的识别算法是否像我们认为的那样出色。整个视觉科学领域都在使用一种称为视觉心理物理学的完全不同的评估机制来研究视觉感知。心理物理学是对受控刺激与实验测试对象所引起的行为反应之间关系的定量研究。心理物理学不是使用汇总统计数据来衡量性能，而是指导我们构建由单个刺激反应组成的项目响应曲线，以找到感知阈值，从而可以确定受试者无法再可靠识别刺激类别的确切点。在本文中，我们引入了一个基于该方法的视觉识别模型的综合评估框架。在数百万个程序渲染的 3D 场景和 2D 图像上，我们比较了著名卷积神经网络的性能。我们的结果对最近提出的类似人类的性能声称提出了质疑，并为纠正新出现的算法缺陷提供了一条途径。
