Temporal variation in natural selection is predicted to strongly impact the evolution and demography of natural populations, with consequences for the rate of adaptation, evolution of plasticity, and extinction risk. Most of the theory underlying these predictions assumes a moving optimum phenotype, with predictions expressed in terms of the temporal variance and autocorrelation of this optimum. However, empirical studies seldom estimate patterns of fluctuations of an optimum phenotype, precluding further progress in connecting theory with observations. To bridge this gap, we assess the evidence for temporal variation in selection on breeding date by modeling a fitness function with a fluctuating optimum, across 39 populations of 21 wild animals, one of the largest compilations of long-term datasets with individual measurements of trait and fitness components. We find compelling evidence for fluctuations in the fitness function, causing temporal variation in the magnitude, but not the direction of selection. However, fluctuations of the optimum phenotype need not directly translate into variation in selection gradients, because their impact can be buffered by partial tracking of the optimum by the mean phenotype. Analyzing individuals that reproduce in consecutive years, we find that plastic changes track movements of the optimum phenotype across years, especially in bird species, reducing temporal variation in directional selection. This suggests that phenological plasticity has evolved to cope with fluctuations in the optimum, despite their currently modest contribution to variation in selection.