1 Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities , Boltzmannstraße 1 , 85748 Garching near Munich , Germany.
2 Poznan Supercomputing and Networking Center , Institute of Bioorganic Chemistry of the Polish Academy of Sciences , ul Z. Noskowskiego 12/14 , 61-704 Poznan , Poland.
Philos Trans A Math Phys Eng Sci. 2019 Apr 8;377(2142):20180151. doi: 10.1098/rsta.2018.0151.
We describe a method for queue wait time prediction in supercomputing clusters. It was designed for use as a part of multi-criteria brokering mechanisms for resource selection in a multi-site High Performance Computing environment. The aim is to incorporate the time jobs stay queued in the scheduling system into the selection criteria. Our method can also be used by the end users to estimate the time to completion of their computing jobs. It uses historical data about the particular system to make predictions. It returns a list of probability estimates of the form ( t , p ), where p is the probability that the job will start before time t . Times t can be chosen more or less freely when deploying the system. Compared to regression methods that only return a single number as a queue wait time estimate (usually without error bars) our prediction system provides more useful information. The probability estimates are calculated using the Bayes theorem with the naive assumption that the attributes describing the jobs are independent. They are further calibrated to make sure they are as accurate as possible, given available data. We describe our service and its REST API and the underlying methods in detail and provide empirical evidence in support of the method's efficacy. This article is part of the theme issue 'Multiscale modelling, simulation and computing: from the desktop to the exascale'.
我们描述了一种用于超级计算集群中排队等待时间预测的方法。它旨在作为多站点高性能计算环境中资源选择的多标准中介机制的一部分使用。其目的是将作业在调度系统中排队的时间纳入选择标准。我们的方法也可以被终端用户用来估计他们的计算作业的完成时间。它使用有关特定系统的历史数据进行预测。它返回一个概率估计列表，形式为 ( t, p )，其中 p 是作业将在时间 t 之前开始的概率。在部署系统时，可以更自由地选择时间 t 。与仅返回单个数字作为队列等待时间估计的回归方法（通常没有误差条）相比，我们的预测系统提供了更有用的信息。概率估计是使用贝叶斯定理计算的，假设描述作业的属性是独立的。进一步对它们进行校准，以确保在可用数据的基础上尽可能准确。我们详细描述了我们的服务及其 REST API 和底层方法，并提供了支持该方法有效性的经验证据。本文是“多尺度建模、模拟和计算：从桌面到 exascale”主题的一部分。
