MCF-SMSIS: Multi-tasking with complementary functions for stereo matching and surgical instrument segmentation.

Stereo matching and instrument segmentation of laparoscopic surgical scenarios are key tasks in robotic surgical automation. Many researchers have been studying the two tasks separately for stereo matching and instrument segmentation. However, the relationship between these two tasks is often neglected. In this paper, we propose a model framework for multi-tasking with complementary functions for stereo matching and surgical instrument segmentation (MCF-SMSIS). We aim to complement the features of instrument prediction segmentation to the parallax matching block of stereo matching. We also propose two new evaluation metrics (MINPD and MAXPD) for assessing how well the parallax range matches the migrated domain when the model used for the stereo matching task undergoes domain migration. We performed stereo matching experiments on the SCARED , SERV-CT dataset as well as instrumentation segmentation experiments on the AutoLaparo dataset. The results demonstrate the effectiveness of the proposed method. In particular, stereo matching supplemented with instrument features reduced EPE, >3px and RMSE Depth in the surgical instrument section by 9.5%, 12.7% and 6.51%, respectively. The instrumentation segmentation performance also achieves a DSC value of 0.9233. Moreover, MCF-SMSIS takes only 0.14 s to infer a set of images. The model code and model weights for each stage are available from https://github.com/wurenkai/MCF-SMSIS.