BACKGROUND: Tooth agenesis, one of the most common developmental anomalies, can affect the function and esthetics of patients. The aim of the present study was to identify genetic clues for familial tooth agenesis and explore the underlying mechanisms, focusing on the role of human dental pulp stem cells (hDPSCs). METHODS: We applied Sanger sequencing to identify the cause of oligodontia in a Chinese family. DNA transfection and functional analysis in DPSCs was also performed to explore the impact of the identified mutation on this phenotype. RESULTS: In this study, a novel frameshift mutation, the twenty-nucleotide deletion (c.128_147del20, p.Met43Serfsx125), in exon1 of MSX1 was detected in a Chinese family causing autosomal dominant nonsyndromic oligodontia. The mutation cosegregated with the tooth agenesis phenotype in this family. DPSCs transfected with mutant MSX1 plasmid showed decreased capacity of osteo/odontogenic differentiation with a lower expression level of dentin sialophosphoprotein (DSPP) and bone sialoprotein (BSP) compared with those transfected with control MSX1 plasmid. Mechanically, control MSX1 showed nuclear localization while the mutant MSX1 inhibited its nuclear translocation and localized on the cytoplasm to inhibit ERK phosphorylation. Furthermore, we inhibited the ERK pathway using ERK inhibitor (U0126) treatment in control MSX1-transfected DPSCs which could downregulate mineralized nodule formation and the expression of odontogenic genes. CONCLUSION: We demonstrated a novel MSX1 mutation causing familial nonsyndromic oligodontia and mechanically MSX1 regulates odontogenesis through the ERK signaling pathway in human dental pulp stem cells.