Introduction Poor physical performance is associated with an increased risk of post-discharge cardiovascular events in patients with heart failure. In this study, we investigated the association between physical frailty and improvements in physical performance through cardiac rehabilitation in hospitalized older patients with heart failure. Methods The study included 100 patients with heart failure (aged ≥65 years) hospitalized between January 2023 and August 2024, with a short physical performance battery (SPPB) score ranging from 1 to 11 points at the initiation of cardiac rehabilitation. Patients achieving an improvement of ≥1 SPPB points during hospitalization were classified as the improved group, while those with unchanged or declining scores comprised the non-improved group. We retrospectively examined the association between physical frailty and improvement in physical performance due to acute-phase cardiac rehabilitation during hospitalization. Results Among the 100 patients, 62 and 38 were categorized into the improved and non-improved groups, respectively. Although no significant differences were observed regarding age, sex, or rehabilitation duration between the groups, the prevalence of physical frailty was significantly higher in the non-improved than in the improved group. Modified Poisson regression analysis, controlling for age, sex, and New York Heart Association functional class, showed that physical frailty was significantly associated with a reduced likelihood of improvement in physical performance. Conclusion Physical frailty may inhibit improvement in physical performance among hospitalized older patients with heart failure. Therefore, enhancing acute-phase rehabilitation strategies for patients with frailty and strengthening the post-discharge follow-up system are essential.