A growing body of evidence suggests that repeated concussions and exposure to repetitive head impacts may be associated with subtle abnormalities in neurological health. Prior studies have demonstrated associations of prior concussion and repetitive head impacts with the brain's microstructure, typically using diffusion magnetic resonance imaging, though the direction of these relationships has varied across groups. Quantitative susceptibility mapping is a quantitative extension of susceptibility weighted imaging that is sensitive to pathophysiology associated with neurotrauma and thus represents an alternative method to characterize the effects of concussion and repetitive head impact exposure on brain microstructure. The goal of this work was to characterize the association of prior concussion and years of contact sport exposure (a proxy of repetitive head impacts) with magnetic susceptibility in a cohort of otherwise healthy male and female collegiate-aged athletes. We hypothesized that concussion and contact sport exposure would be independently associated with lower susceptibility in white matter regions. Higher general symptom severity and psychological symptoms were observed in athletes with a greater history of concussion, but not years of contact sport exposure. No associations between concussion or years of exposure with cognitive performance were observed. Voxel-wise analyses found that individuals with more prior concussions also had lower magnetic susceptibility in two predominantly white matter clusters including the superior longitudinal fasciculi and forceps major. No associations of susceptibility and contact sport exposure were observed. Finally, lower susceptibility in the identified regions was associated with worse psychological symptoms, worse general symptoms, and worse performance on a composite measure of fluid cognition tasks. Current results suggest that more prior concussions in otherwise healthy collegiate-aged athletes are associated with decreases in susceptibility that are in turn associated with elevated symptom reporting and poorer cognitive performance.