Wang Han, Ji Xiang, Jin Lei, Ji Yujiao, Wang Guangcheng
School of Transportation and Civil Engineering, Nantong University, Nantong 226019, China.
Sensors (Basel). 2024 Jul 18;24(14):4654. doi: 10.3390/s24144654.
With the popularity of smartphones, a large number of "phubbers" have emerged who are engrossed in their phones regardless of the situation. In response to the potential dangers that phubbers face while traveling, this paper proposes a multimodal danger perception network model and early warning system for phubbers, designed for mobile devices. This proposed model consists of surrounding environment feature extraction, user behavior feature extraction, and multimodal feature fusion and recognition modules. The environmental feature module utilizes MobileNet as the backbone network to extract environmental description features from the rear-view image of the mobile phone. The behavior feature module uses acceleration time series as observation data, maps the acceleration observation data to a two-dimensional image space through GADFs (Gramian Angular Difference Fields), and extracts behavior description features through MobileNet, while utilizing statistical feature vectors to enhance the representation capability of behavioral features. Finally, in the recognition module, the environmental and behavioral characteristics are fused to output the type of hazardous state. Experiments indicate that the accuracy of the proposed model surpasses existing methods, and it possesses the advantages of compact model size (28.36 Mb) and fast execution speed (0.08 s), making it more suitable for deployment on mobile devices. Moreover, the developed image-acceleration multimodal phubber hazard recognition network combines the behavior of mobile phone users with surrounding environmental information, effectively identifying potential hazards for phubbers.
随着智能手机的普及，出现了大量“低头族”，他们无论在何种情况下都全神贯注于手机。针对低头族在出行时面临的潜在危险，本文提出了一种面向移动设备的低头族多模态危险感知网络模型及预警系统。该模型由周围环境特征提取、用户行为特征提取以及多模态特征融合与识别模块组成。环境特征模块利用MobileNet作为骨干网络，从手机后视图像中提取环境描述特征。行为特征模块以加速度时间序列作为观测数据，通过格拉姆角差分场（Gramian Angular Difference Fields，GADFs）将加速度观测数据映射到二维图像空间，并通过MobileNet提取行为描述特征，同时利用统计特征向量增强行为特征的表示能力。最后，在识别模块中，将环境特征和行为特征进行融合，输出危险状态类型。实验表明，所提模型的准确率超过现有方法，且具有模型尺寸紧凑（28.36 Mb）和执行速度快（0.08 s）的优点，更适合在移动设备上部署。此外，所开发的图像 - 加速度多模态低头族危险识别网络将手机用户的行为与周围环境信息相结合，有效识别低头族的潜在危险。
