Centre d'Analyse et de Mathématique Sociales, UMR 8557 CNRS-EHESS, École des Hautes Études en Sciences Sociales, 75006 Paris, France
Centre d'Analyse et de Mathématique Sociales, UMR 8557 CNRS-EHESS, École des Hautes Études en Sciences Sociales, 75006 Paris, France, and Laboratoire de Physique de l'ENS, Sorbonne Université, Université de Paris, UMR 8023 CNRS-ENS, École Normale Supérieure, 75006 Paris, France
Neural Comput. 2022 Jan 14;34(2):437-475. doi: 10.1162/neco_a_01454.
Classification is one of the major tasks that deep learning is successfully tackling. Categorization is also a fundamental cognitive ability. A well-known perceptual consequence of categorization in humans and other animals, categorical perception, is notably characterized by a within-category compression and a between-category separation: two items, close in input space, are perceived closer if they belong to the same category than if they belong to different categories. Elaborating on experimental and theoretical results in cognitive science, here we study categorical effects in artificial neural networks. We combine a theoretical analysis that makes use of mutual and Fisher information quantities and a series of numerical simulations on networks of increasing complexity. These formal and numerical analyses provide insights into the geometry of the neural representation in deep layers, with expansion of space near category boundaries and contraction far from category boundaries. We investigate categorical representation by using two complementary approaches: one mimics experiments in psychophysics and cognitive neuroscience by means of morphed continua between stimuli of different categories, while the other introduces a categoricality index that, for each layer in the network, quantifies the separability of the categories at the neural population level. We show on both shallow and deep neural networks that category learning automatically induces categorical perception. We further show that the deeper a layer, the stronger the categorical effects. As an outcome of our study, we propose a coherent view of the efficacy of different heuristic practices of the dropout regularization technique. More generally, our view, which finds echoes in the neuroscience literature, insists on the differential impact of noise in any given layer depending on the geometry of the neural representation that is being learned, that is, on how this geometry reflects the structure of the categories.
分类是深度学习成功解决的主要任务之一。分类也是一种基本的认知能力。人类和其他动物分类的一个著名的感知后果是范畴知觉，其显著特点是类内压缩和类间分离：两个在输入空间中接近的项目，如果它们属于同一类别，那么它们的感知就会比属于不同类别的项目更接近。本文在认知科学的实验和理论结果的基础上，研究了人工神经网络中的范畴效应。我们结合了一种理论分析，该分析利用了互信息和 Fisher 信息量，并对越来越复杂的网络进行了一系列数值模拟。这些形式和数值分析提供了对深层神经网络表示的几何结构的深入了解，在类别边界附近的空间扩展，而在远离类别边界的地方则收缩。我们通过两种互补的方法来研究范畴表示：一种方法通过不同类别之间的刺激连续变化来模拟心理物理学和认知神经科学的实验，另一种方法引入了一个范畴性指数，该指数对网络中的每个层量化了在神经群体水平上类别可分离性。我们在浅层和深层神经网络上都表明，类别学习自动诱导范畴知觉。我们进一步表明，网络中的层次越深，范畴效应越强。作为我们研究的结果，我们提出了一种对 dropout 正则化技术的不同启发式实践的有效性的一致看法。更一般地说，我们的观点在神经科学文献中找到了回声，强调了在任何给定层中噪声的不同影响取决于正在学习的神经表示的几何形状，即这种几何形状如何反映类别结构。
