It is widely accepted that listening to music improves subjective feelings and reduces fatigue sensations, and different kinds of music lead to different activations of these feelings. Recently, cardiac autonomic nervous modulation has been proposed as a useful objective indicator of fatigue. However, scientific considerations of the relation between feelings of fatigue and cardiac autonomic nervous modulation while listening to music are still lacking. In this study, we examined which subjective feelings of fatigue are related to participants' cardiac autonomic nervous function while they listen to music. We used an album of comfortable and relaxing environmental music, with blended sounds from a piano and violin as well as natural sound sources. We performed a crossover trial of environmental music and silent sessions for 20 healthy subjects, 12 females, and 8 males, after their daily work shift. We measured changes in eight types of subjective feelings, including healing, fatigue, sleepiness, relaxation, and refreshment, using the KOKORO scale, a subjective mood measurement system for self-reported feelings. Further, we obtained measures of cardiac autonomic nervous function on the basis of heart rate variability before and after the sessions. During the music session, subjective feelings significantly shifted toward healing and a secure/relaxed feeling and these changes were greater than those in the silent session. Heart rates (ΔHR) in the music session significantly decreased compared with those in the silent session. Other cardiac autonomic parameters such as high-frequency (HF) component and the ratio of low-frequency (LF) and HF components (LF/HF) were similar in the two sessions. In the linear regression analysis of the feelings with ΔHR and changes in LF/HF (ΔLF/HF), increases and decreases in ΔHR were correlated to the feeling axes of Fatigue-Healing and Anxiety/Tension-Security/Relaxation, whereas those in ΔLF/HF were related to the feeling axes of Sleepiness-Wakefulness and Gloomy-Refreshed. This indicated that listening to music improved the participants' feelings of fatigue and decreased their heart rates. However, it did not reduce the cardiac LF/HF, suggesting that cardiac LF/HF might show a delayed response to fatigue. Thus, we demonstrated changes in cardiac autonomic nervous functions based on feelings of fatigue.