Bao Shunxing, Lee Ho Hin, Yang Qi, Remedios Lucas W, Deng Ruining, Cui Can, Cai Leon Y, Xu Kaiwen, Yu Xin, Chiron Sophie, Li Yike, Patterson Nathan Heath, Wang Yaohong, Li Jia, Liu Qi, Lau Ken S, Roland Joseph T, Coburn Lori A, Wilson Keith T, Landman Bennett A, Huo Yuankai
Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA.
Department of Computer Science, Vanderbilt University, Nashville, TN, USA.
Proc Mach Learn Res. 2024;227:1406-1422.
Multiplex immunofluorescence (MxIF) is an advanced molecular imaging technique that can simultaneously provide biologists with multiple (i.e., more than 20) molecular markers on a single histological tissue section. Unfortunately, due to imaging restrictions, the more routinely used hematoxylin and eosin (H&E) stain is typically unavailable with MxIF on the same tissue section. As biological H&E staining is not feasible, previous efforts have been made to obtain H&E whole slide image (WSI) from MxIF via deep learning empowered virtual staining. However, the tiling effect is a long-lasting problem in high-resolution WSI-wise synthesis. The MxIF to H&E synthesis is no exception. Limited by computational resources, the cross-stain image synthesis is typically performed at the patch-level. Thus, discontinuous intensities might be visually identified along with the patch boundaries assembling all individual patches back to a WSI. In this work, we propose a deep learning based unpaired high-resolution image synthesis method to obtain virtual H&E WSIs from MxIF WSIs (each with 27 markers/stains) with reduced tiling effects. Briefly, we first extend the CycleGAN framework by adding simultaneous nuclei and mucin segmentation supervision as spatial constraints. Then, we introduce a random walk sliding window shifting strategy during the optimized inference stage, to alleviate the tiling effects. The validation results show that our spatially constrained synthesis method achieves a 56% performance gain for the downstream cell segmentation task. The proposed inference method reduces the tiling effects by using 50% fewer computation resources without compromising performance. The proposed random sliding window inference method is a plug-and-play module, which can be generalized for other high-resolution WSI image synthesis applications. The source code with our proposed model are available at https://github.com/MASILab/RandomWalkSlidingWindow.git.
多重免疫荧光（MxIF）是一种先进的分子成像技术，它可以在单个组织学组织切片上同时为生物学家提供多种（即超过20种）分子标记。不幸的是，由于成像限制，在同一组织切片上，MxIF通常无法同时使用更常用的苏木精和伊红（H&E）染色。由于生物H&E染色不可行，之前已经有人尝试通过深度学习赋能的虚拟染色从MxIF中获取H&E全切片图像（WSI）。然而，在高分辨率的全切片图像合成中，拼接效应一直是个长期存在的问题。从MxIF合成H&E也不例外。受计算资源限制，交叉染色图像合成通常在补丁级别进行。因此，在将所有单独的补丁拼接回全切片图像时，沿着补丁边界可能会在视觉上识别出不连续的强度。在这项工作中，我们提出了一种基于深度学习的无配对高分辨率图像合成方法，以从MxIF全切片图像（每个图像有27种标记/染色）中获取虚拟H&E全切片图像，减少拼接效应。简而言之，我们首先通过添加同时进行的细胞核和黏液分割监督作为空间约束来扩展CycleGAN框架。然后，我们在优化推理阶段引入随机游走滑动窗口移动策略，以减轻拼接效应。验证结果表明，我们的空间约束合成方法在下游细胞分割任务中实现了56%的性能提升。所提出的推理方法在不影响性能的情况下，使用的计算资源减少了50%，从而减少了拼接效应。所提出的随机滑动窗口推理方法是一个即插即用模块，可以推广到其他高分辨率全切片图像合成应用中。我们提出的模型的源代码可在https://github.com/MASILab/RandomWalkSlidingWindow.git获取。
