Understanding how species' thermal limits have evolved across the tree of life is central to predicting species' responses to climate change. Here, using experimentally-derived estimates of thermal tolerance limits for over 2000 terrestrial and aquatic species, we show that most of the variation in thermal tolerance can be attributed to a combination of adaptation to current climatic extremes, and the existence of evolutionary 'attractors' that reflect either boundaries or optima in thermal tolerance limits. Our results also reveal deep-time climate legacies in ectotherms, whereby orders that originated in cold paleoclimates have presently lower cold tolerance limits than those with warm thermal ancestry. Conversely, heat tolerance appears unrelated to climate ancestry. Cold tolerance has evolved more quickly than heat tolerance in endotherms and ectotherms. If the past tempo of evolution for upper thermal limits continues, adaptive responses in thermal limits will have limited potential to rescue the large majority of species given the unprecedented rate of contemporary climate change.