BACKGROUND: Silymarin, a blend of flavonolignans isolated from plant Silybum marianum L., has long been used as an herbal medicine. Biogenic routes especially the plant-based synthesis of selenium nanoparticles (SeNPs) is safe, eco-friendly, nontoxic and being considered as one of the best strategies for treatment of cancer. PURPOSE: Silymarin-mediated green synthesis of SeNPs and their possibility as an anticancer agent have not been reported to date. Therefore, our present study was aimed to synthesize and characterize the selenium mediated silymarin nanoparticles (Si-SeNPs) from silymarin and investigate their possibility as an anticancer agent. METHODS: The physicochemical characteristics of Si-SeNPs were analyzed using various analytical techniques, such as HPLC, field emission-transmission electron microscope, energy-dispersive X-ray spectrometer, and thermogravimetric analysis. The underlying molecular mechanism were evaluated using AGS gastric cancer cells. RESULTS: Compared with silymarin, the Si-SeNPs exhibited significantly increased cytotoxic effect of AGS cells without exhibiting toxicity on normal cells. Real time PCR and western blotting analysis indicated that Si-SeNPs induced expression of Bax/Bcl-2, cytochrome c, and cleavage of caspase proteins, which is associated with mitochondria-mediated apoptosis signaling in AGS cells. Moreover, agonist assay using PI3K activator indicated that Si-SeNPs-inhibited PI3K/AKT/mTOR pathways were significantly associated as an autophagy and apoptosis signaling in AGS cells. CONCLUSION: Our study demonstrated the improved anticancer efficacy of Si-SeNPs- induced apoptosis and autophagy pathways, and therefore recommended Si-SeNPs as a novel anticancer agent after in vivo studies.