Bethge Hans Lukas, Weisheit Inga, Dortmund Mauritz Sandro, Landes Timm, Zabic Miroslav, Linde Marcus, Debener Thomas, Heinemann Dag
Institute of Horticultural Production Systems, Department of Phytophotonics, Leibniz University Hannover, Herrenhäuser Str. 2, 30419, Hannover, Germany.
Hannover Centre for Optical Technologies, Leibniz University Hannover, Nienburger Straße 17, 30167, Hannover, Germany.
Plant Methods. 2024 Nov 17;20(1):175. doi: 10.1186/s13007-024-01296-y.
The early and specific detection of abiotic and biotic stresses, particularly their combinations, is a major challenge for maintaining and increasing plant productivity in sustainable agriculture under changing environmental conditions. Optical imaging techniques enable cost-efficient and non-destructive quantification of plant stress states. Monomodal detection of certain stressors is usually based on non-specific/indirect features and therefore is commonly limited in their cross-specificity to other stressors. The fusion of multi-domain sensor systems can provide more potentially discriminative features for machine learning models and potentially provide synergistic information to increase cross-specificity in plant disease detection when image data are fused at the pixel level.
In this study, we demonstrate successful multi-modal image registration of RGB, hyperspectral (HSI) and chlorophyll fluorescence (ChlF) kinetics data at the pixel level for high-throughput phenotyping of A. thaliana grown in Multi-well plates and an assay with detached leaf discs of Rosa × hybrida inoculated with the black spot disease-inducing fungus Diplocarpon rosae. Here, we showcase the effects of (i) selection of reference image selection, (ii) different registrations methods and (iii) frame selection on the performance of image registration via affine transform. In addition, we developed a combined approach for registration methods through NCC-based selection for each file, resulting in a robust and accurate approach that sacrifices computational time. Since image data encompass multiple objects, the initial coarse image registration using a global transformation matrix exhibited heterogeneity across different image regions. By employing an additional fine registration on the object-separated image data, we achieved a high overlap ratio. Specifically, for the A. thaliana test set, the overlap ratios (OR) were 98.0 ± 2.3% for RGB-to-ChlF and 96.6 ± 4.2% for HSI-to-ChlF. For the Rosa × hybrida test set, the values were 98.9 ± 0.5% for RGB-to-ChlF and 98.3 ± 1.3% for HSI-to-ChlF.
The presented multi-modal imaging pipeline enables high-throughput, high-dimensional phenotyping of different plant species with respect to various biotic or abiotic stressors. This paves the way for in-depth studies investigating the correlative relationships of the multi-domain data or the performance enhancement of machine learning models via multi modal image fusion.
在不断变化的环境条件下，早期且特异性地检测非生物和生物胁迫，尤其是它们的组合，是可持续农业中维持和提高作物生产力面临的一项重大挑战。光学成像技术能够以具有成本效益且无损的方式量化植物胁迫状态。对某些胁迫源的单模态检测通常基于非特异性/间接特征，因此在区分其他胁迫源方面通常存在局限性。当在像素级别融合图像数据时，多域传感器系统的融合可以为机器学习模型提供更多潜在的判别特征，并可能提供协同信息以提高植物病害检测中的交叉特异性。
在本研究中，我们展示了在像素级别成功实现RGB、高光谱（HSI）和叶绿素荧光（ChlF）动力学数据的多模态图像配准，用于对生长在多孔板中的拟南芥进行高通量表型分析，以及对接种了诱发黑斑病的真菌蔷薇双壳菌的杂交玫瑰离体叶片进行检测。在这里，我们展示了（i）参考图像选择、（ii）不同配准方法和（iii）帧选择对通过仿射变换进行图像配准性能的影响。此外，我们通过对每个文件基于归一化互相关（NCC）的选择开发了一种组合配准方法，从而得到一种牺牲计算时间但稳健且准确的方法。由于图像数据包含多个对象，使用全局变换矩阵进行的初始粗图像配准在不同图像区域表现出异质性。通过对分离对象的图像数据进行额外的精细配准，我们实现了高重叠率。具体而言，对于拟南芥测试集，RGB与ChlF的重叠率（OR）为98.0±2.3%，HSI与ChlF的重叠率为96.6±4.2%。对于杂交玫瑰测试集，RGB与ChlF的值为98.9±0.5%，HSI与ChlF的值为98.3±1.3%。
所提出的多模态成像流程能够针对各种生物或非生物胁迫源对不同植物物种进行高通量、高维表型分析。这为深入研究多域数据的相关关系或通过多模态图像融合提高机器学习模型的性能铺平了道路。
