Benchmarking the operation of quantum heuristics and Ising machines: scoring parameter setting strategies on optimization applications.

We discuss guidelines for evaluating the performance of parameterized stochastic solvers for optimization problems, with particular attention to systems that employ novel hardware, such as digital quantum processors running variational algorithms, analog processors performing quantum annealing, or coherent Ising machines. We illustrate through an example a benchmarking procedure grounded in the statistical analysis of the expectation of a given performance metric measured in a test environment. In particular, we discuss the necessity and cost of setting parameters that affect the algorithm's performance. The optimal value of these parameters could vary significantly between instances of the same target problem. We present an open-source software package that facilitates the design, evaluation, and visualization of practical parameter tuning strategies for the complex use of the heterogeneous components of the solver. We examine in detail an example using parallel tempering and a simulator of a photonic coherent Ising machine computing and display the scoring of an illustrative baseline family of parameter setting strategies that feature an exploration-exploitation trade-off.