To evaluate the tumor targeting efficiency of self-assembled polymeric nanoparticles, four glycol chitosan nanoparticles (CNPs) with different degrees of hydrophobic substitution were prepared by coupling 7.5, 12, 23, and 35 wt.% of 5β-cholanic acid to hydrophilic glycol chitosan polymer (GC). The sizes and zeta-potentials of different CNPs in aqueous condition were not significantly different, but their stability and deformability were greatly dependent upon the degree of substitution (DS) of 5β-cholanic acid. With an increase in hydrophobicity, CNPs became more stable and rigid, as characterized by SDS-PAGE and filtration tests. To compare with CNPs, linear GC and polystyrene nanoparticles (PSNPs) were employed as controls. In vivo tumor accumulation of Cy5.5-labeled linear GC, polystyrene nanoparticles (PSNPs) and CNPs were monitored in flank tumors and liver tumor-bearing mice models using near-infrared fluorescence (NIRF) imaging systems. CNPs displayed higher tumor accumulation than GC and PSNPs via the enhanced permeability and retention (EPR) effect. Interestingly, CNPs containing 23 wt.% of 5β-cholanic acid (CNP-23%) showed the highest tumor-targeting efficiency compared to other CNPs. As exemplified in this study, the stability of CNP-23% is better than CNP-7.5% and CNP-12% containing 7.5 wt.% and 12 wt.% of 5β-cholanic acid, respectively, and the deformability of CNP-23% is better than that of CNP-35% containing 35 wt.% of 5β-cholanic acid. We proposed that the superior tumor-targeting efficiency of CNP-23% is mainly due to their balanced stability and deformability in vivo. This study demonstrates that the degree of hydrophobic substitution of self-assembled nanoparticles could determine their stability and deformability. Importantly, they were founded to be the key factors which affect their tumor-targeting efficiency in vivo, and so that these factors should be highly considered during developing nanoparticles for tumor-targeted imaging or drug delivery.