Opt Express. 2022 Oct 24;30(22):39520-39533. doi: 10.1364/OE.470324.
A lensless camera is an imaging system that uses a mask in place of a lens, making it thinner, lighter, and less expensive than a lensed camera. However, additional complex computation and time are required for image reconstruction. This work proposes a deep learning model named Raw3dNet that recognizes hand gestures directly on raw videos captured by a lensless camera without the need for image restoration. In addition to conserving computational resources, the reconstruction-free method provides privacy protection. Raw3dNet is a novel end-to-end deep neural network model for the recognition of hand gestures in lensless imaging systems. It is created specifically for raw video captured by a lensless camera and has the ability to properly extract and combine temporal and spatial features. The network is composed of two stages: 1. spatial feature extractor (SFE), which enhances the spatial features of each frame prior to temporal convolution; 2. 3D-ResNet, which implements spatial and temporal convolution of video streams. The proposed model achieves 98.59% accuracy on the Cambridge Hand Gesture dataset in the lensless optical experiment, which is comparable to the lensed-camera result. Additionally, the feasibility of physical object recognition is assessed. Further, we show that the recognition can be achieved with respectable accuracy using only a tiny portion of the original raw data, indicating the potential for reducing data traffic in cloud computing scenarios.
无透镜相机是一种成像系统，它使用掩模代替透镜，使其比透镜相机更薄、更轻、更便宜。然而，图像重建需要额外的复杂计算和时间。这项工作提出了一种名为 Raw3dNet 的深度学习模型，它可以直接在无透镜相机拍摄的原始视频上识别手势，而无需图像恢复。除了节省计算资源外，这种无需重建的方法还提供了隐私保护。Raw3dNet 是一种用于无透镜成像系统中手势识别的新型端到端深度神经网络模型。它是专门为无透镜相机拍摄的原始视频创建的，具有正确提取和组合时间和空间特征的能力。该网络由两个阶段组成：1. 空间特征提取器（SFE），在进行时间卷积之前增强每一帧的空间特征；2. 3D-ResNet，对视频流进行空间和时间卷积。在所进行的无透镜光学实验中，该模型在剑桥手姿数据集上达到了 98.59%的准确率，与透镜相机的结果相当。此外，还评估了物理对象识别的可行性。进一步表明，仅使用原始原始数据的一小部分就可以实现可接受的识别准确率，这表明在云计算场景中减少数据流量的潜力。
