The application of Fe3O4 nanoparticles (NPs) has made great progress in the diagnosis of disease and in the drug delivery system for cancer therapy, but the relative mechanisms of potential toxicity induced by Fe3O4 have not kept pace with its development in the application, which has hampered its further clinical application. In this article, we used two kinds of human hepatoma cell lines, SK-Hep-1 and Hep3B, to investigate the cytotoxic effects and the involved mechanisms of small Fe3O4 NPs with different diameters (6 nm, 9 nm, and 14 nm). Results showed that the size of NPs effectively influences the cytotoxicity of hepatoma cells: 6 nm Fe3O4 NPs exhibited negligible cytotoxicity and 9 nm Fe3O4 NPs affected cytotoxicity via cellular mitochondrial dysfunction and by inducing necrosis mediated through the mitochondria-dependent intracellular reactive oxygen species generation. Meanwhile, 14 nm Fe3O4 NPs induced cytotoxicity by impairing the integrity of plasma membrane and promoting massive lactate dehydrogenase leakage. These results explain the detailed mechanism of different diameters of small Fe3O4 NPs-induced cytotoxicity. We anticipate that this study will provide different insights into the cytotoxicity mechanism of Fe3O4 NPs, so as to make them safer to use in clinical application.