Research Center for Mathematics and Interdisciplinary Sciences, Frontiers Science Center for Nonlinear Expectations (Ministry of Education), Shandong University, Shandong 266000, China.
BioMap, Inc., Beijing 100086, China.
Bioinformatics. 2022 Oct 14;38(20):4797-4805. doi: 10.1093/bioinformatics/btac566.
Serial-section electron microscopy (ssEM) is a powerful technique for cellular visualization, especially for large-scale specimens. Limited by the field of view, a megapixel image of whole-specimen is regularly captured by stitching several overlapping images. However, suffering from distortion by manual operations, lens distortion or electron impact, simple rigid transformations are not adequate for perfect mosaic generation. Non-linear deformation usually causes 'ghosting' phenomenon, especially with high magnification. To date, existing microscope image processing tools provide mature rigid stitching methods but have no idea with local distortion correction.
In this article, following the development of unsupervised deep learning, we present a multi-scale network to predict the dense deformation fields of image pairs in ssEM and blend these images into a clear and seamless montage. The model is composed of two pyramidal backbones, sharing parameters and interacting with a set of registration modules, in which the pyramidal architecture could effectively capture large deformation according to multi-scale decomposition. A novel 'intermediate-space solving' paradigm is adopted in our model to treat inputted images equally and ensure nearly perfect stitching of the overlapping regions. Combining with the existing rigid transformation method, our model further improves the accuracy of sequential image stitching. Extensive experimental results well demonstrate the superiority of our method over the other traditional methods.
The code is available at https://github.com/HeracleBT/ssEM_stitching.
Supplementary data are available at Bioinformatics online.
连续切片电子显微镜（ssEM）是一种强大的细胞可视化技术，尤其适用于大规模样本。受视场限制，通常通过拼接几个重叠的图像来捕获整个样本的百万像素图像。然而，由于手动操作、镜头失真或电子冲击引起的失真，简单的刚性变换不足以实现完美的镶嵌生成。非线性变形通常会导致“重像”现象，尤其是在高倍放大时。迄今为止，现有的显微镜图像处理工具提供了成熟的刚性拼接方法，但没有局部变形校正的概念。
在本文中，我们紧跟无监督深度学习的发展，提出了一种多尺度网络，用于预测 ssEM 中图像对的密集变形场，并将这些图像融合成清晰无缝的拼接图。该模型由两个金字塔形骨干组成，共享参数并与一组配准模块相互作用，其中金字塔结构可以根据多尺度分解有效地捕获大变形。我们的模型采用了一种新颖的“中间空间求解”范例，平等对待输入的图像，并确保重叠区域的近乎完美拼接。结合现有的刚性变换方法，我们的模型进一步提高了顺序图像拼接的准确性。广泛的实验结果充分证明了我们的方法优于其他传统方法。
代码可在 https://github.com/HeracleBT/ssEM_stitching 上获得。
补充数据可在 Bioinformatics 在线获得。
