Biomedical Informatics Group (GIB), Escuela Técnica Superior de Ingenieros Informáticos, Universidad Politécnica de Madrid, Campus de Montegancedo S/N, Madrid 28660, Spain.
Biomedical Informatics Group (GIB), Escuela Técnica Superior de Ingenieros Informáticos, Universidad Politécnica de Madrid, Campus de Montegancedo S/N, Madrid 28660, Spain.
Comput Methods Programs Biomed. 2021 Apr;202:105958. doi: 10.1016/j.cmpb.2021.105958. Epub 2021 Feb 2.
Nanoparticles present properties that can be applied to a wide range of fields such as biomedicine, electronics or optics. The type of properties depends on several characteristics, being some of them related with the particle structure. A proper characterization of nanoparticles is crucial since it could affect their applications. To characterize a particle shape and size, the nanotechnologists employ Electron Microscopy (EM) to obtain images of nanoparticles and perform measures over them. This task could be tedious, repetitive and slow, we present a Deep Learning method based on Convolutional Neural Networks (CNNs) to detect, segment, infer orientations and reconstruct microscope images of nanoparticles. Since machine learning algorithms depend on annotated data and there is a lack of annotated datasets of nanoparticles, our work makes use of artificial datasets of images resembling real nanoparticles photographs.
Our work is divided into three tasks. Firstly, a method to create annotated datasets of artificial images resembling Scanning Electron Microscope (SEM). Secondly, two models of convolutional neural networks are trained using the artificial datasets previously generated, the first one is in charge of the detection and segmentation of the nanoparticles while the second one will infer the nanoparticle orientation. Finally, the 3D reconstruction module will recreate in a 3D scene the set of detected particles.
We have tested our method with five different shapes of basic nanoparticles: spheres, cubes, ellipsoids, hexagonal discs and octahedrons. An analysis of the reconstructions was conducted by manually comparing each of them with the real images. The results obtained have been promising, the particles are segmented and reconstructed accordingly to their shapes and orientations.
We have developed a method for nanoparticle detection and segmentation in microscope images. Moreover, we can also infer an approximation of the 3D orientation of the particles and, in conjunction with the detections, create a 3D reconstruction of the photographs. The novelty of our approximation lies in the dataset used. Instead of using annotated images, we have created the datasets simulating the microscope images by using basic geometrical objects that imitate real nanoparticles.
纳米粒子具有多种性质，可应用于生物医药、电子或光学等多个领域。这些性质取决于多个特征，其中一些与粒子结构有关。对纳米粒子进行适当的特性分析至关重要，因为这可能会影响它们的应用。为了对粒子的形状和大小进行特征分析，纳米技术人员采用电子显微镜（EM）获取纳米粒子的图像，并对其进行测量。这项任务可能既繁琐、重复又耗时，因此我们提出了一种基于卷积神经网络（CNN）的深度学习方法，用于检测、分割、推断纳米粒子的方向并重建显微镜图像。由于机器学习算法依赖于标注数据，而纳米粒子的标注数据集又很缺乏，因此我们的工作利用了类似于真实纳米粒子照片的人工图像数据集。
我们的工作分为三个任务。首先，我们开发了一种方法，用于创建类似于扫描电子显微镜（SEM）的人工图像的标注数据集。其次，我们使用之前生成的人工数据集训练了两个卷积神经网络模型，第一个模型负责纳米粒子的检测和分割，第二个模型则用于推断纳米粒子的方向。最后，3D 重建模块将在 3D 场景中重新创建检测到的粒子集。
我们使用五种不同形状的基本纳米粒子（球体、立方体、椭球体、六边盘和八面体）对我们的方法进行了测试。通过手动比较每个重建结果与真实图像，对重建结果进行了分析。得到的结果很有前景，纳米粒子根据其形状和方向进行了分割和重建。
我们开发了一种用于显微镜图像中纳米粒子检测和分割的方法。此外，我们还可以推断出粒子的大致 3D 方向，并结合检测结果，创建照片的 3D 重建。我们的方法的新颖之处在于使用的数据集。我们没有使用标注图像，而是通过使用基本的几何形状来创建模拟显微镜图像的数据集，这些基本的几何形状模仿真实的纳米粒子。
