Torres-Sospedra Joaquín, Jiménez Antonio R, Moreira Adriano, Lungenstrass Tomás, Lu Wei-Chung, Knauth Stefan, Mendoza-Silva Germán Martín, Seco Fernando, Pérez-Navarro Antoni, Nicolau Maria João, Costa António, Meneses Filipe, Farina Joaquín, Morales Juan Pablo, Lu Wen-Chen, Cheng Ho-Ti, Yang Shi-Shen, Fang Shih-Hau, Chien Ying-Ren, Tsao Yu
Institute of New Imaging Technologies, Universitat Jaume I, 12071 Castelló, Spain.
Centre for Automation and Robotics (CAR), CSIC-UPM, 28500 Arganda del Rey, Spain.
Sensors (Basel). 2018 Feb 6;18(2):487. doi: 10.3390/s18020487.
The development of indoor positioning solutions using smartphones is a growing activity with an enormous potential for everyday life and professional applications. The research activities on this topic concentrate on the development of new positioning solutions that are tested in specific environments under their own evaluation metrics. To explore the real positioning quality of smartphone-based solutions and their capabilities for seamlessly adapting to different scenarios, it is needed to find fair evaluation frameworks. The design of competitions using extensive pre-recorded datasets is a valid way to generate open data for comparing the different solutions created by research teams. In this paper, we discuss the details of the 2017 IPIN indoor localization competition, the different datasets created, the teams participating in the event, and the results they obtained. We compare these results with other competition-based approaches (Microsoft and Perf-loc) and on-line evaluation web sites. The lessons learned by organising these competitions and the benefits for the community are addressed along the paper. Our analysis paves the way for future developments on the standardization of evaluations and for creating a widely-adopted benchmark strategy for researchers and companies in the field.
利用智能手机开发室内定位解决方案是一项不断发展的活动，在日常生活和专业应用中具有巨大潜力。关于这一主题的研究活动集中于开发新的定位解决方案，并在特定环境下根据其自身的评估指标进行测试。为了探究基于智能手机的解决方案的实际定位质量及其无缝适应不同场景的能力，需要找到公平的评估框架。利用大量预先录制的数据集设计竞赛是生成开放数据以比较研究团队创建的不同解决方案的有效方式。在本文中，我们讨论了2017年IPIN室内定位竞赛的细节、创建的不同数据集、参赛团队以及他们取得的结果。我们将这些结果与其他基于竞赛的方法（微软和Perf-loc）以及在线评估网站进行比较。本文还阐述了组织这些竞赛所吸取的经验教训以及对社区的益处。我们的分析为评估标准化的未来发展以及为该领域的研究人员和公司创建广泛采用的基准策略铺平了道路。
