A well-known natural ingredient found in several medicinal plants, berberine (Ber), has been shown to have anticancer properties against a range of malignancies. The limited solubility and bioavailability of berberine can be addressed using Ber-loaded nanoparticles. In this study, we compared the in vitro cytotoxic effects of both Ber-loaded silver nanoparticles (Ber-AgNPs) and Ber-loaded selenium nanoparticles (Ber-SeNPs) in the human liver cancer cell line (HepG2) and mouse normal liver cells (BNL). The IC values in HepG2 for berberine, Ber-AgNPs, Ber-SeNPs, and cisplatin were 26.69, 1.16, 0.04, and 0.33 µg/mL, respectively. Our results show that Ber and its Ag and Se nanoparticles exerted a good antitumor effect against HepG2 cells by inducing apoptosis via upregulating p53, Bax, cytosolic cytochrome C levels, and caspase-3 activity, and the down-regulation of Bcl-2 levels. Similarly, incubation with Ber and both Ber-NPs (Ag and Se) led to a significant dose-dependent elevation in inflammatory markers' (TNF-α, NF-κB, and COX-2) levels compared to the control group. In addition, it led to the arrest of the G1 cell cycle by depleting the expression of and mRNA. Furthermore, Ber and both Ber-NPs (Ag and Se) caused a significant dose-dependent increase in LDH activity in HepG2 cells. Furthermore, our findings offer evidence that Ber and its nanoparticles intensified oxidative stress in HepG2 cells. Furthermore, the migration rate of cells subjected to berberine and its nanoforms was notably decreased compared to that of control cells. It can be inferred that Ber nanoparticles exhibited superior anticancer efficacy against HepG2 compared to unprocessed Ber, perhaps due to their improved solubility and bioavailability. Furthermore, Ber-SeNPs exhibited greater efficacy than Ber-AgNPs, possibly as a result of the inherent anticancer characteristics of selenium.