How evolutionary forces interact to maintain genetic variation within populations has been a matter of extensive theoretical debates. While mutation and exogenous gene flow increase genetic variation, stabilizing selection and genetic drift are expected to deplete it. To date, levels of genetic variation observed in natural populations are hard to predict without accounting for other processes, such as balancing selection in heterogeneous environments. We aimed to empirically test three hypotheses: (i) admixed populations have higher quantitative genetic variation due to introgression from other gene pools, (ii) quantitative genetic variation is lower in populations from harsher environments (i.e., experiencing stronger selection), and (iii) quantitative genetic variation is higher in populations from heterogeneous environments. Using growth, phenological and functional trait data from three clonal common gardens and 33 populations (522 clones) of maritime pine (Pinus pinaster Aiton), we estimated the association between the population-specific total genetic variances (i.e., among-clone variances) for these traits and ten population-specific indices related to admixture levels (estimated based on 5165 SNPs), environmental temporal and spatial heterogeneity and climate harshness. Populations experiencing colder winters showed consistently lower genetic variation for early height growth (a fitness-related trait in forest trees) in the three common gardens. Within-population quantitative genetic variation was not associated with environmental heterogeneity or population admixture for any trait. Our results provide empirical support for the potential role of natural selection in reducing genetic variation for early height growth within populations, which indirectly gives insight into the adaptive potential of populations to changing environments.