Li Bin, Chen Peijun, Liu Hongfu, Guo Weisi, Cao Xianbin, Du Junzhao, Zhao Chenglin, Zhang Jun
School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing, China.
School of Information and Electronics, Beijing Institute of Technology, Beijing, China.
Nat Comput Sci. 2021 Mar;1(3):221-228. doi: 10.1038/s43588-021-00039-6. Epub 2021 Mar 25.
Despite the great potential of deep neural networks (DNNs), they require massive weights and huge computational resources, creating a vast gap when deploying artificial intelligence at low-cost edge devices. Current lightweight DNNs, achieved by high-dimensional space pre-training and post-compression, present challenges when covering the resources deficit, making tiny artificial intelligence hard to be implemented. Here we report an architecture named random sketch learning, or Rosler, for computationally efficient tiny artificial intelligence. We build a universal compressing-while-training framework that directly learns a compact model and, most importantly, enables computationally efficient on-device learning. As validated on different models and datasets, it attains substantial memory reduction of ~50-90× (16-bits quantization), compared with fully connected DNNs. We demonstrate it on low-cost hardware, whereby the computation is accelerated by >180× and the energy consumption is reduced by ~10×. Our method paves the way for deploying tiny artificial intelligence in many scientific and industrial applications.
尽管深度神经网络（DNN）具有巨大潜力，但它们需要大量权重和巨大的计算资源，这在将人工智能部署到低成本边缘设备时造成了巨大差距。当前通过高维空间预训练和后压缩实现的轻量级DNN，在弥补资源不足方面面临挑战，使得微型人工智能难以实现。在此，我们报告一种名为随机草图学习（Rosler）的架构，用于实现计算高效的微型人工智能。我们构建了一个通用的训练时压缩框架，该框架直接学习紧凑模型，最重要的是，能够实现计算高效的设备端学习。在不同模型和数据集上的验证表明，与全连接DNN相比，它可实现约50 - 90倍的显著内存减少（16位量化）。我们在低成本硬件上进行了演示，计算速度加快了180倍以上，能耗降低了约10倍。我们的方法为在许多科学和工业应用中部署微型人工智能铺平了道路。
