Brain age is a metric that quantifies the degree of aging of a brain based on whole-brain anatomical characteristics. While associations between individual human brain regions and environmental or genetic factors have been investigated, how brain age is associated with those factors remains unclear. We investigated these associations using UK Biobank data. We first trained a statistical model for obtaining relative brain age (RBA), a metric describing a subject's brain age relative to peers, based on whole-brain anatomical measurements, from training set subjects (n = 5,193). We then applied this model to evaluation set subjects (n = 12,115) and tested the association of RBA with tobacco smoking, alcohol consumption, and genetic variants. We found that daily or almost daily consumption of tobacco and alcohol were both significantly associated with increased RBA (P < 0.001). We also found SNPs significantly associated with RBA (p-value < 5E-8). The SNP most significantly associated with RBA is located in MAPT gene. Our results suggest that both environmental and genetic factors are associated with structural brain aging.