Triboelectric Mat Multimodal Sensing System (TMMSS) Enhanced by Infrared Image Perception for Sleep and Emotion-Relevant Activity Monitoring.

To implement digital-twin smart home applications, the mat sensing system based on triboelectric sensors is commonly used for gait information collection from daily activities. Yet traditional mat sensing systems often miss upper body motions and fail to adequately project these into the virtual realm, limiting their specific application scenarios. Herein, triboelectric mat multimodal sensing system is designed, enhanced with a commercial infrared imaging sensor, to capture diverse sensory information for sleep and emotion-relevant activity monitoring without compromising privacy. This system generates pixel-based area ratio mappings across the entire mat array, solely based on the integral operation of triboelectric outputs. Additionally, it utilizes multimodal sensory intelligence and deep-learning analytics to detect different sleeping postures and monitor comprehensive sleep behaviors and emotional states associated with daily activities. These behaviors are projected into the metaverse, enhancing virtual interactions. This multimodal sensing system, cost-effective and non-intrusive, serves as a functional interface for diverse digital-twin smart home applications such as healthcare, sports monitoring, and security.