ESCAN: Efficient GPU sharing for cascade neural network inference.

Cascading, as a multi-model combination approach, balances model execution efficiency and accuracy. This excellent method is widely used in various industrial production and commercial deployments, particularly in cloud-based inference services. With the increasing demand for low-latency services, researchers are more focused on the execution efficiency of these models, especially device utilization. It is highly desirable to fully utilize GPU resources by multiplexing different inference tasks on the same GPU through device-sharing techniques such as Multiprocessing Services (MPS). However, we find it struggling when applying MPS to cascade neural networks consisting of multiple related submodels. These difficulties arise primarily from the early-exit mechanism and the execution order of the submodels. To address these obstacles, we analyzed the characteristics of cascade neural networks and combined them with device-sharing optimization techniques. Our findings indicate that improving the efficiency of cascade models through device sharing requires a balance between the gains from sharing devices and the potential wastage of computation resources due to the early-exit mechanism. Based on our analysis, we proposed ESCAN, a GPU-sharing optimization framework for online inference of cascade neural networks. This framework includes exit-ratio-aware batch-parallel execution strategies and the corresponding resource allocation algorithms, all integrated into PyTorch. Experiments show that ESCAN improves inference efficiency by an average of 19.53% compared to the execution strategy with all cascade submodels running in parallel. Additionally, ESCAN significantly improves the efficiency of searching for computation resource allocation schemes. ESCAN optimizes the utilization of computational resources through effective GPU-sharing, greatly enhancing the efficiency of online inference for cascade models. This approach delivers a low-latency, high-precision optimization solution for interactive online services based on cascade neural networks.