The automatic detection of arrhythmia is of primary importance due to the huge number of victims caused worldwide by cardiovascular diseases. To this aim, several deep learning approaches have been recently proposed to automatically classify heartbeats in a small number of classes. Most of these approaches use convolutional neural networks (CNNs), exploiting some bi-dimensional representation of the ECG signal, such as spectrograms, scalograms, or similar. However, by adopting such representations, state-of-the-art approaches usually rely on the magnitude information alone, while the important phase information is often neglected. Motivated by these considerations, the focus of this paper is aimed at investigating the effect of fusing the magnitude and phase of the continuous wavelet transform (CWT), known as the scalogram and phasogram, respectively. Scalograms and phasograms are fused in a simple CNN-based architecture by using several fusion strategies, which fuse the information in the input layer, some intermediate layers, or in the output layer. Numerical results evaluated on the PhysioNet MIT-BIH Arrhythmia database show the effectiveness of the proposed ideas. Although a simple architecture is used, their competitiveness is high compared to other state-of-the-art approaches, by obtaining an overall accuracy of about 98.5% and sensitivity and specificity of 98.5% and 95.6%, respectively.