The development of numerical and arithmetic abilities constitutes a crucial cornerstone in our modern and educated societies. Difficulties to acquire these central skills can lead to severe consequences for an individual's well-being and nation's economy. In the present review, we describe our current broad understanding of the functional and structural brain organization that supports the development of numbers and arithmetic. The existing evidence points towards a complex interaction among multiple domain-specific (e.g., representation of quantities and number symbols) and domain-general (e.g., working memory, visual-spatial abilities) cognitive processes, as well as a dynamic integration of several brain regions into functional networks that support these processes. These networks are mainly, but not exclusively, located in regions of the frontal and parietal cortex, and the functional and structural dynamics of these networks differ as a function of age and performance level. Distinctive brain activation patterns have also been shown for children with dyscalculia, a specific learning disability in the domain of mathematics. Although our knowledge about the developmental brain dynamics of number and arithmetic has greatly improved over the past years, many questions about the interaction and the causal involvement of the abovementioned functional brain networks remain. This review provides a broad and critical overview of the known developmental processes and what is yet to be discovered.