Tooth agenesis is a congenital anomaly frequently seen in humans. Several genes have been associated with non-syndromic tooth agenesis, including msh homeobox 1 (MSX1), paired box 9 (PAX9), axis inhibition protein 2 (AXIN2), ectodysplasin A (EDA), and wingless-type MMTV integration site family member 10A (WNT10A). In this study, we investigated a Chinese family with non-syndromic tooth agenesis. A novel missense mutation (c.C1978T) in AXIN2 was identified in affected members. The mutation results in a His660Tyr substitution located between the Axin beta-catenin binding domain and the DIX domain of the axis inhibition protein 2 (AXIN2). We analysed this novel AXIN2 mutant, together with two reported AXIN2 mutants [c.1966C>T (p.Arg656Stop) and c.1994delG (p.Leu688Stop)] that cause colorectal cancer with and without oligodontia, to study the effect of the mutant p.His660Tyr on the Wnt/β-catenin signaling pathway and to compare the molecular pathogenesis of different AXIN2 mutants in tooth agenesis and carcinogenesis. Further in vitro experiments indicated that the mutant p.His660Tyr caused inhibition of the Wnt/β-catenin pathway, and the mutants p.Arg656Stop and p.Leu688Stop resulted in over-activation of the Wnt/β-catenin pathway. In line with previous AXIN2 mutation studies, we suggest that AXIN2 mutations with different levels of severity may have distinct effects on the Wnt pathway and the phenotype of disease. Our study provides functional evidence supporting the notion that both inhibition and over-activation of the Wnt pathway may lead to tooth agenesis.