The homeobox, msh-like 1 (MSX1) protein is essential for cell proliferation and differentiation. Tooth germ development of Msx1 knockout mouse is arrested at the bud stage, impeding an understanding of its role beyond this stage of tooth development. The aims of this study were to investigate the potential role of MSX1 in the regulation of proliferation and differentiation of dental mesenchymal cells in culture, and to preliminarily explore its underlying mechanism of action. Tooth germs were isolated from embryonic day (E)15.5 mice. The mesenchyme was separated and digested into a single-cell suspension, and then cultured in vitro. Isolated dental mesenchymal cells were transfected with MSX1 small interfering RNA, and the effects on cell proliferation, cell cycle distribution, and the expression of bone morphogenetic protein 2 (Bmp2) and bone morphogenetic protein 4 (Bmp4) were studied. We also compared the expression levels of alkaline phosphatase (Alp), type I collagen (Col1A), osteocalcin (Ocn), runt-related transcription factor 2 (Runx2), dentin sialophosphoprotein (Dspp) and dentin matrix protein 1 (Dmp1), and mineralized nodule formation, between control and MSX1 siRNA-transfected groups after the induction of odontoblast differentiation. Knockdown of Msx1 expression was associated with decreased cell proliferation, prolonged time in the S phase of the cell cycle, enhanced odontoblast differentiation, and elevated Bmp2 and Bmp4 expression. We conclude that MSX1 may promote proliferation and prevent the differentiation of dental mesenchymal cells by the inhibition of Bmp2 and Bmp4 expression.