Department of Psychology, Stanford University, Stanford, California, United States of America.
Wu Tsai Neurosciences Institute, Stanford University, Stanford, California, United States of America.
PLoS Comput Biol. 2022 Jan 7;18(1):e1009739. doi: 10.1371/journal.pcbi.1009739. eCollection 2022 Jan.
Task-optimized convolutional neural networks (CNNs) show striking similarities to the ventral visual stream. However, human-imperceptible image perturbations can cause a CNN to make incorrect predictions. Here we provide insight into this brittleness by investigating the representations of models that are either robust or not robust to image perturbations. Theory suggests that the robustness of a system to these perturbations could be related to the power law exponent of the eigenspectrum of its set of neural responses, where power law exponents closer to and larger than one would indicate a system that is less susceptible to input perturbations. We show that neural responses in mouse and macaque primary visual cortex (V1) obey the predictions of this theory, where their eigenspectra have power law exponents of at least one. We also find that the eigenspectra of model representations decay slowly relative to those observed in neurophysiology and that robust models have eigenspectra that decay slightly faster and have higher power law exponents than those of non-robust models. The slow decay of the eigenspectra suggests that substantial variance in the model responses is related to the encoding of fine stimulus features. We therefore investigated the spatial frequency tuning of artificial neurons and found that a large proportion of them preferred high spatial frequencies and that robust models had preferred spatial frequency distributions more aligned with the measured spatial frequency distribution of macaque V1 cells. Furthermore, robust models were quantitatively better models of V1 than non-robust models. Our results are consistent with other findings that there is a misalignment between human and machine perception. They also suggest that it may be useful to penalize slow-decaying eigenspectra or to bias models to extract features of lower spatial frequencies during task-optimization in order to improve robustness and V1 neural response predictivity.
任务优化卷积神经网络（CNNs）与腹侧视觉流表现出惊人的相似性。然而，人类无法察觉的图像干扰会导致 CNN 做出错误的预测。通过研究对图像干扰具有鲁棒性或不具有鲁棒性的模型的表示，我们深入了解了这种脆弱性。理论表明，系统对这些干扰的鲁棒性可能与神经网络响应集合的特征谱的幂律指数有关，其中幂律指数更接近且大于一的系统对输入干扰的敏感性更低。我们表明，老鼠和猕猴初级视觉皮层（V1）中的神经反应服从该理论的预测，其特征谱的幂律指数至少为一。我们还发现，模型表示的特征谱的衰减速度比神经生理学观察到的要慢，并且鲁棒模型的特征谱衰减速度略快，幂律指数高于非鲁棒模型。特征谱的缓慢衰减表明，模型响应中的大量方差与精细刺激特征的编码有关。因此，我们研究了人工神经元的空间频率调谐，发现它们中的很大一部分更喜欢高空间频率，并且鲁棒模型的空间频率分布更接近猕猴 V1 细胞的测量空间频率分布。此外，鲁棒模型是比非鲁棒模型更好的 V1 模型。我们的结果与其他发现一致，即人类和机器感知之间存在不匹配。它们还表明，在任务优化过程中，可以通过惩罚缓慢衰减的特征谱或偏向模型提取较低空间频率的特征来提高鲁棒性和 V1 神经反应可预测性，这可能是有用的。
