Reeb Rachel A, Aziz Naeem, Lapp Samuel M, Kitzes Justin, Heberling J Mason, Kuebbing Sara E
Department of Biological Sciences, University of Pittsburgh, Pittsburgh, PA, United States.
Section of Botany, Carnegie Museum of Natural History, Pittsburgh, PA, United States.
Front Plant Sci. 2022 Jan 17;12:787407. doi: 10.3389/fpls.2021.787407. eCollection 2021.
Community science image libraries offer a massive, but largely untapped, source of observational data for phenological research. The iNaturalist platform offers a particularly rich archive, containing more than 49 million verifiable, georeferenced, open access images, encompassing seven continents and over 278,000 species. A critical limitation preventing scientists from taking full advantage of this rich data source is labor. Each image must be manually inspected and categorized by phenophase, which is both time-intensive and costly. Consequently, researchers may only be able to use a subset of the total number of images available in the database. While iNaturalist has the potential to yield enough data for high-resolution and spatially extensive studies, it requires more efficient tools for phenological data extraction. A promising solution is automation of the image annotation process using deep learning. Recent innovations in deep learning have made these open-source tools accessible to a general research audience. However, it is unknown whether deep learning tools can accurately and efficiently annotate phenophases in community science images. Here, we train a convolutional neural network (CNN) to annotate images of into distinct phenophases from iNaturalist and compare the performance of the model with non-expert human annotators. We demonstrate that researchers can successfully employ deep learning techniques to extract phenological information from community science images. A CNN classified two-stage phenology (flowering and non-flowering) with 95.9% accuracy and classified four-stage phenology (vegetative, budding, flowering, and fruiting) with 86.4% accuracy. The overall accuracy of the CNN did not differ from humans ( = 0.383), although performance varied across phenophases. We found that a primary challenge of using deep learning for image annotation was not related to the model itself, but instead in the quality of the community science images. Up to 4% of images in iNaturalist were taken from an improper distance, were physically manipulated, or were digitally altered, which limited both human and machine annotators in accurately classifying phenology. Thus, we provide a list of photography guidelines that could be included in community science platforms to inform community scientists in the best practices for creating images that facilitate phenological analysis.
社区科学图像库为物候研究提供了大量但基本未被利用的观测数据来源。iNaturalist平台提供了一个特别丰富的档案库，包含超过4900万张可验证、地理参考的开放获取图像，涵盖七大洲和超过27.8万种物种。阻碍科学家充分利用这一丰富数据源的一个关键限制因素是人力。每张图像都必须由物候阶段进行人工检查和分类，这既耗时又昂贵。因此，研究人员可能只能使用数据库中可用图像总数的一个子集。虽然iNaturalist有潜力产生足够的数据用于高分辨率和空间广泛的研究，但它需要更有效的物候数据提取工具。一个有前途的解决方案是使用深度学习实现图像标注过程的自动化。深度学习的最新创新使这些开源工具可供一般研究人员使用。然而，尚不清楚深度学习工具是否能够准确、高效地标注社区科学图像中的物候阶段。在这里，我们训练了一个卷积神经网络（CNN）来将来自iNaturalist的图像标注为不同的物候阶段，并将该模型的性能与非专业人类标注者进行比较。我们证明研究人员可以成功地采用深度学习技术从社区科学图像中提取物候信息。一个CNN对两阶段物候（开花和不开花）的分类准确率为95.9%，对四阶段物候（营养、萌芽、开花和结果）的分类准确率为86.4%。CNN的总体准确率与人类没有差异（P = 0.383），尽管在不同物候阶段性能有所不同。我们发现，使用深度学习进行图像标注的一个主要挑战与模型本身无关，而是与社区科学图像的质量有关。iNaturalist中高达4%的图像是从不当距离拍摄的、经过物理处理的或经过数字修改的，这限制了人类和机器标注者准确分类物候。因此，我们提供了一份摄影指南清单，可纳入社区科学平台，以告知社区科学家创建有助于物候分析的图像的最佳实践。
