Hepatocellular carcinoma (HCC) is a malignancy for which no effective drugs are available. Paxilline is derived from an endophytic fungus of the leaves of Baphicacanthus cusia (Nees) Bremek. In a previous study, we found that paxilline inhibited the proliferation of HepG2 cells; however, its mechanism remains unclear. In this study, ESI and NMR were used to characterize the paxilline structure. Network pharmacology analysis was performed with databases and software to obtain the core targets and signaling pathways associated with the anti-HCC effects of paxilline. Molecular docking was performed to validate the preliminary affinity of paxilline for the core targets. For further in vitro experiments, a CCK8 assay was performed to detect cell viability, a wound healing assay was performed to detect cell migration, an Annexin V-FITC assay was performed to detect the cell cycle and apoptosis rate in HepG2 cells, RT-qPCR analysis was performed to detect the expression of cell cycle-related genes and autophagy-related genes, Immunofluorescence staining was performed to detect the expression of LC3B, and Western blotting was performed to detect the expression of apoptosis-related proteins and autophagy-related proteins. As a result, we obtained a white powder, which was identified as paxilline. Network analysis and molecular docking results revealed that apoptosis-related and autophagy-relatted protein were key targets (mTOR and PI3K) for paxilline anti-HCC. Further examination revealed that paxilline promoted HepG2 cell apoptosis, inhibited HepG2 cell migration, and arrested HepG2 cell in the S phage. RT-qPCR analysis revealed that paxilline markedly downregulated the mRNA expression of Cyclin D1, CDK4, LC3B, mTOR, Parkin, and PINK1. Immunofluorescence staining demonstrated a significant upregulation of LC3B protein expression following paxilline treatment. Further validation by Western blotting showed that paxilline significantly increased the expression of LC3B II/I, bax, cleaved-PARP, and cleaved-caspase 3, while significantly decreased the expression of bcl-2. Additionally, a significant promotion of cellular apoptosis and expression of apoptotic proteins when treatment with chloroquine (CQ)/rapamycin (Rapa). Meanwhile, when combined with paxilline, it was found that paxilline may have a synergistic effects with Rapa, jointly promoting cellular apoptosis and the expression of proapoptotic proteins. In conclusion, these findings reveled paxilline alleviates HCC through autophagy-mediated apoptosis.