Zuckerman Mind Brain Behavior Institute, Columbia University, 3227 Broadway, New York, NY, 10027, USA.
Sci Rep. 2023 Sep 1;13(1):14375. doi: 10.1038/s41598-023-40807-0.
Deep neural network models (DNNs) are essential to modern AI and provide powerful models of information processing in biological neural networks. Researchers in both neuroscience and engineering are pursuing a better understanding of the internal representations and operations that undergird the successes and failures of DNNs. Neuroscientists additionally evaluate DNNs as models of brain computation by comparing their internal representations to those found in brains. It is therefore essential to have a method to easily and exhaustively extract and characterize the results of the internal operations of any DNN. Many models are implemented in PyTorch, the leading framework for building DNN models. Here we introduce TorchLens, a new open-source Python package for extracting and characterizing hidden-layer activations in PyTorch models. Uniquely among existing approaches to this problem, TorchLens has the following features: (1) it exhaustively extracts the results of all intermediate operations, not just those associated with PyTorch module objects, yielding a full record of every step in the model's computational graph, (2) it provides an intuitive visualization of the model's complete computational graph along with metadata about each computational step in a model's forward pass for further analysis, (3) it contains a built-in validation procedure to algorithmically verify the accuracy of all saved hidden-layer activations, and (4) the approach it uses can be automatically applied to any PyTorch model with no modifications, including models with conditional (if-then) logic in their forward pass, recurrent models, branching models where layer outputs are fed into multiple subsequent layers in parallel, and models with internally generated tensors (e.g., injections of noise). Furthermore, using TorchLens requires minimal additional code, making it easy to incorporate into existing pipelines for model development and analysis, and useful as a pedagogical aid when teaching deep learning concepts. We hope this contribution will help researchers in AI and neuroscience understand the internal representations of DNNs.
深度神经网络模型（DNN）是现代人工智能的基础，为生物神经网络中的信息处理提供了强大的模型。神经科学和工程领域的研究人员都在努力更好地理解 DNN 的成功和失败所依赖的内部表示和操作。神经科学家还通过将 DNN 的内部表示与大脑中的表示进行比较，将 DNN 评估为大脑计算的模型。因此，必须有一种方法可以方便、详尽地提取和描述任何 DNN 的内部操作结果。许多模型都是用 PyTorch 实现的，PyTorch 是构建 DNN 模型的领先框架。在这里，我们引入了 TorchLens，这是一个新的开源 Python 包，用于提取和描述 PyTorch 模型中的隐藏层激活。与解决此问题的现有方法相比，TorchLens 具有以下特点：（1）它详尽地提取了所有中间操作的结果，而不仅仅是与 PyTorch 模块对象相关的结果，从而生成了模型计算图中每一步的完整记录；（2）它提供了模型完整计算图的直观可视化以及模型正向传递中每个计算步骤的元数据，以便进一步分析；（3）它包含一个内置的验证过程，可自动验证所有保存的隐藏层激活的准确性；（4）它使用的方法可以自动应用于任何 PyTorch 模型，而无需修改，包括在正向传递中具有条件（if-then）逻辑、递归模型、分支模型（其中层输出并行输入到多个后续层）以及具有内部生成张量的模型（例如，注入噪声）。此外，使用 TorchLens 只需添加很少的额外代码，因此很容易将其集成到现有的模型开发和分析管道中，并且在教授深度学习概念时也很有用。我们希望这项贡献将帮助人工智能和神经科学领域的研究人员理解 DNN 的内部表示。
