The aim of this paper is to study the anti-tumor mechanism of volatile oil from Thunb. (sodium new houttuyfonate, SNH). , SNH exhibited a concentration-dependent cytotoxic effect against four human cancer lines (HepG2, A2780, MCF-7, SKOV-3). SNH treatment with different concentrations induced HepG2 cells to exhibit varying degrees of morphological changes in apoptotic features, such as round shape, cell shrinkage and formation of apoptotic body. It was observed that SNH caused the decrease in Bcl-2 mRNA expression and triggered the apoptosis of HepG2 cells. Wound healing assay and RT-PCR results showed that the decrease in the expression level of MMP9 and VEGF was observed in HepG2 cells after treatment with SNH for 48 h, suggesting that the extracellular matrix pathway degradation was involved in the HepG2 cells migration. Moreover, we got an insight into the binding mode of SNH into the MMP9 active site through 3D pharmacophore models. Docking study and molecular dynamics (MD) simulation analysis sheds light on that SNH was completely embedded into the MMP9 active site and formed hydrogen bonds with key catalytic residues of MMP9, including Ala191, His190, Ala189 and Glu227. The prediction of SNH binding interaction energies in the MMP9 was almost in good agreement with the original inhibitor EN140. experiments, both SNH and cyclophosphamide significantly reduced tumor weights and their tumor inhibitory rates were 50.78% and 82.61% respectively. This study demonstrated that SNH was an apoptosis inducer in HepG2 cells. SNH has four possible functions, that it could induce apoptosis by mitochondria pathway in HepG2 cells, inhibit the tumor growth, regulate Bcl-2 family mRNA expression and effectively subdue migration of hepatocellular carcinoma cells by decreasing the expression of MMP9 and VEGF. Therefore, SNH might be a potential candidate drug for the treatment of hepatocellular carcinoma, which could provide a reference for further clinical research.