A novel nanoscale molecular probe is formulated in order to reduce toxicity and side effects of antitumor drug doxorubicin (DOX) in normal tissues and to enhance the detection sensitivity during early imaging diagnosis. The mechanism involves a specific targeting of Arg-Gly-Asp peptide (RGD)-GX1 heterogeneous dimer peptide-conjugated dendrigraft poly-l-lysine (DGL)-magnetic nanoparticle (MNP) composite by αβ-integrin/vasculature endothelium receptor-mediated synergetic effect. The physicochemical properties of the nanoprobe were characterized by using transmission electron microscope, Fourier transform infrared spectroscopy, X-ray diffraction, dynamic light scattering (DLS), and vibrating sample magnetometer. The average diameter of the resulting MNP-DGL-RGD-GX1-DOX nanoparticles (NPs) was ~150-160 nm by DLS under simulate physiological medium. In the present experimental system, the loading amount of DOX on NPs accounted for 414.4 mg/g for MNP-DGL-RGD-GX1-DOX. The results of cytotoxicity, flow cytometry, and cellular uptake consistently indicated that the MNP-DGL-RGD-GX1-DOX NPs were inclined to target HepG2 cells in selected three kinds of cells. In vitro exploration of molecular mechanism revealed that cell apoptosis was associated with the overexpression of Fas protein and the significant activation of caspase-3. In vivo magnetic resonance imaging and biodistribution study showed that the MNP-DGL-RGD-GX1-DOX formulation had high affinity to the tumor tissue, leading to more aggregation of NPs in the tumor. In vivo antitumor efficacy research verified that MNP-DGL-RGD-GX1-DOX NPs possessed significant antitumor activity and the tumor inhibitory rate reached 78.5%. These results suggested that NPs could be promising in application to early diagnosis and therapy in hepatocellular carcinoma as a specific nanoprobe.