Is mathematical language similar to natural language? Are language areas used by mathematicians when they do mathematics? And does the brain comprise a generic semantic system that stores mathematical knowledge alongside knowledge of history, geography or famous people? Here, we refute those views by reviewing three functional MRI studies of the representation and manipulation of high-level mathematical knowledge in professional mathematicians. The results reveal that brain activity during professional mathematical reflection spares perisylvian language-related brain regions as well as temporal lobe areas classically involved in general semantic knowledge. Instead, mathematical reflection recycles bilateral intraparietal and ventral temporal regions involved in elementary number sense. Even simple fact retrieval, such as remembering that 'the sine function is periodical' or that 'London buses are red', activates dissociated areas for math versus non-math knowledge. Together with other fMRI and recent intracranial studies, our results indicated a major separation between two brain networks for mathematical and non-mathematical semantics, which goes a long way to explain a variety of facts in neuroimaging, neuropsychology and developmental disorders.This article is part of a discussion meeting issue 'The origins of numerical abilities'.