It is quite generally assumed that the overdamped Langevin equation provides a quantitative description of the dynamics of a classical Brownian particle in the long time limit. We establish and investigate a paradigm anomalous diffusion process governed by an underdamped Langevin equation with an explicit time dependence of the system temperature and thus the diffusion and damping coefficients. We show that for this underdamped scaled Brownian motion (UDSBM) the overdamped limit fails to describe the long time behaviour of the system and may practically even not exist at all for a certain range of the parameter values. Thus persistent inertial effects play a non-negligible role even at significantly long times. From this study a general questions on the applicability of the overdamped limit to describe the long time motion of an anomalously diffusing particle arises, with profound consequences for the relevance of overdamped anomalous diffusion models. We elucidate our results in view of analytical and simulations results for the anomalous diffusion of particles in free cooling granular gases.