Structures suppressed during evolution can be retraced due to atavisms and vestiges. Atavism is an exceptional emergence of an ancestral form in a living individual. In contrast, ancestral vestige regularly occurs in all members of an actual species. We surveyed data about the vestigial and atavistic teeth in mammals, updated them by recent findings in mouse and human embryos, and discussed their ontogenetic and evolutionary implications. In the mouse incisor and diastema regions, dental placodes are transiently distinct being morphologically similar to the early tooth primordia in reptiles. Two large vestigial buds emerge in front of the prospective first molar and presumably correspond to the premolars eliminated during mouse evolution. The incorporation of the posterior premolar vestige into the lower first molar illustrates the putative mechanism of evolutionary disappearance of the last premolar in the mice. In mutant mice, devious development of the ancestral tooth primordia might lead to their revivification and origin of atavistic supernumerary teeth. Similarity in the developmental schedule between three molars in mice and the respective third and fourth deciduous premolar and the first molar in humans raises a question about putative homology of these teeth. The complex patterning of the vestibular and dental epithelium in human embryos is reminiscent of the pattern of "Zahnreihen" in lower vertebrates. A hypothesis was presented about the developmental relationship between the structures at the external aspect of the dentition in mammals (oral vestibule, pre-lacteal teeth, paramolar cusps/teeth), the tooth glands in reptiles, and the earliest teeth in lower vertebrates.