Ren Yixiang, Ye Zhenhui, Song Guanghua, Jiang Xiaohong
School of Aeronautics and Astronautics, Zhejiang University, Hangzhou 310027, China.
College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China.
Entropy (Basel). 2022 May 1;24(5):638. doi: 10.3390/e24050638.
Mobile crowdsensing (MCS) is attracting considerable attention in the past few years as a new paradigm for large-scale information sensing. Unmanned aerial vehicles (UAVs) have played a significant role in MCS tasks and served as crucial nodes in the newly-proposed space-air-ground integrated network (SAGIN). In this paper, we incorporate SAGIN into MCS task and present a (SAG-MCS) problem. Based on multi-source observations from embedded sensors and satellites, an aerial UAV swarm is required to carry out energy-efficient data collection and recharging tasks. Up to date, few studies have explored such multi-task MCS problem with the cooperation of UAV swarm and satellites. To address this multi-agent problem, we propose a novel deep reinforcement learning (DRL) based method called (ms-SDRGN). Our ms-SDRGN approach incorporates a multi-scale convolutional encoder to process multi-source raw observations for better feature exploitation. We also use a graph attention mechanism to model inter-UAV communications and aggregate extra neighboring information, and utilize a gated recurrent unit for long-term performance. In addition, a stochastic policy can be learned through a maximum-entropy method with an adjustable temperature parameter. Specifically, we design a heuristic reward function to encourage the agents to achieve global cooperation under partial observability. We train the model to convergence and conduct a series of case studies. Evaluation results show statistical significance and that ms-SDRGN outperforms three state-of-the-art DRL baselines in SAG-MCS. Compared with the best-performing baseline, ms-SDRGN improves 29.0% reward and 3.8% CFE score. We also investigate the scalability and robustness of ms-SDRGN towards DRL environments with diverse observation scales or demanding communication conditions.
在过去几年中，移动群智感知（MCS）作为一种大规模信息感知的新范式，正吸引着广泛关注。无人机（UAV）在MCS任务中发挥了重要作用，并成为新提出的空天地一体化网络（SAGIN）中的关键节点。在本文中，我们将SAGIN纳入MCS任务，并提出了一个（SAG - MCS）问题。基于嵌入式传感器和卫星的多源观测，需要一个空中无人机群来执行节能数据收集和充电任务。到目前为止，很少有研究探讨无人机群与卫星合作的这种多任务MCS问题。为了解决这个多智能体问题，我们提出了一种基于深度强化学习（DRL）的新颖方法，称为（ms - SDRGN）。我们的ms - SDRGN方法结合了一个多尺度卷积编码器来处理多源原始观测，以更好地利用特征。我们还使用图注意力机制对无人机间通信进行建模，并聚合额外的相邻信息，并利用门控循环单元来实现长期性能。此外，可以通过具有可调温度参数的最大熵方法学习随机策略。具体来说，我们设计了一个启发式奖励函数，以鼓励智能体在部分可观测性下实现全局合作。我们将模型训练至收敛，并进行了一系列案例研究。评估结果显示出统计学意义，并且ms - SDRGN在SAG - MCS中优于三个最先进的DRL基线。与表现最佳的基线相比，ms - SDRGN的奖励提高了29.0%，CFE分数提高了3.8%。我们还研究了ms - SDRGN在具有不同观测尺度或苛刻通信条件的DRL环境中的可扩展性和鲁棒性。
