Graphene oxide derivatives have shown promise for photothermal cancer therapy due to their efficient light-to-heat conversion in the near-infrared (NIR) range. Therefore, in this study, we investigated the potential of newly synthesized pristine (nGO) and PEGylated (nGO-PEG) graphene oxide nanoparticles, for photothermal therapy of hepatocellular carcinoma (HepG2) cells. We evaluated various aspects of cellular behavior, including migration, growth, morphology, cell membranes integrity, mitochondrial dynamics, actin cytoskeleton organization, and ROS generation along with the expression of genes linked to apoptosis (CASP8, BAX), autophagy (BECN1), cell cycle arrest (CDKN1A), and metastasis (HMMR). Our findings reveal that 5 min of 808 nm NIR irradiation caused a mild temperature increase enhancing cytotoxicity, with nGO showing higher toxicity by disrupting cell morphology, reducing proliferation, and increasing ROS levels. In contrast, nGO-PEG more effectively suppressed cell motility and demonstrated improved biocompatibility. Gene expression analysis revealed upregulation of apoptosis-related genes in nGO-PEG-treated cells indicating mitochondrial damage, while nGO induced autophagy, as seen by increased BECN1 expression. The findings point to distinct therapeutic potentials: nGO as a potent cytotoxic agent inducing autophagy, and nGO-PEG as a more biocompatible nanoparticle promoting apoptosis. This dual-pathway analysis provides a basis for tailored therapeutic strategies for liver cancer.