With age, our blood vessels are prone to aging, which induces cardiovascular disease. As an important basis for diagnosing heart disease and evaluating heart function, the electrocardiogram (ECG) records cardiac physiological electrical activity. Abnormalities in cardiac physiological activity are directly reflected in the ECG. Thus, ECG research is conducive to heart disease diagnosis. Considering the complexity of arrhythmia detection, we present an improved convolutional neural network (CNN) model for accurate classification. Compared with the traditional machine learning methods, CNN requires no additional feature extraction steps due to the automatic feature processing layers. In this paper, an improved CNN is proposed to automatically classify the heartbeat of arrhythmia. Firstly, all the heartbeats are divided from the original signals. After segmentation, the ECG heartbeats can be inputted into the first convolutional layers. In the proposed structure, kernels with different sizes are used in each convolution layer, which takes full advantage of the features in different scales. Then a max-pooling layer followed. The outputs of the last pooling layer are merged and as the input to fully-connected layers. Our experiment is in accordance with the AAMI inter-patient standard, which included normal beats (N), supraventricular ectopic beats (S), ventricular ectopic beats (V), fusion beats (F), and unknown beats (Q). For verification, the MIT arrhythmia database is introduced to confirm the accuracy of the proposed method, then, comparative experiments are conducted. The experiment demonstrates that our proposed method has high performance for arrhythmia detection, the accuracy is 99.06%. When properly trained, the proposed improved CNN model can be employed as a tool to automatically detect different kinds of arrhythmia from ECG.