This paper investigates the interactions between human serum albumin (HSA) and CdTe quantum dots (QDs) with nearly identical hydrodynamic size, but capped with four different ligands (MPA, NAC, and GSH are negatively charged; CA is positively charged) under physiological conditions. The investigation was carried out using fluorescence spectroscopy, circular dichroism (CD) spectra, UV-vis spectroscopy, and dynamic light scattering (DLS). The results of fluorescence quenching and UV-vis absorption spectra experiments indicated the formation of the complex of HSA and negatively charged QDs (MPA-CdTe, NAC-CdTe, and GSH-CdTe), which was also reconfirmed by the increasing of the hydrodynamic radius of QDs. The K(a) values of the three negatively charged QDs are of the same order of magnitude, indicating that the interactions are related to the nanoparticle itself rather than the ligands. ΔH<0 and ΔS>0 implied that the electrostatic interactions play predominant roles in the adsorption process. Furthermore, it was also proven that QDs can induce the conformational changes of HSA from the CD spectra and the three-dimensional fluorescence spectra of HSA. However, our results demonstrate that the interaction mechanism between the positively charged QDs (CA-CdTe) and HSA is significantly different from negatively charged QDs. For CA-CdTe QDs, both the static and dynamic quenching occur within the investigated range of concentrations. According to the DLS results, some large-size agglomeration also emerged.