Machine learning methods are increasingly used in cardiovascular research. In order to highlight opportunities and challenges of the evaluation of studies applying machine learning, we use examples from cardiac electrophysiology, a field characterized by large and often imbalanced amounts of data. We provide recommendations and guidance on evaluating and presenting supervised machine learning studies. We recommend proper cohort selection, keeping training and testing data strictly separate, and comparing results to a reference model without machine learning as basic principles to ensure the quality of studies using machine learning methods. We furthermore recommend specific metrics and plots when reporting on machine learning including on models for multi-channel time series or images. This Best Practice paper represents a possible blueprint to help evaluate machine learning-based medical tests in cardiac electrophysiology and beyond.