Quantum dots (QDs) are recognized as some of the most promising candidates for future applications in biomedicine. However, concerns about their safety have delayed their widespread application. Human serum albumin (HSA) is the main protein component of the circulatory system. It is important to explore the interaction of QDs with HSA for the potential in vivo application of QDs. Herein, using spectroscopy and isothermal titration calorimetry (ITC), the effect of glutathione-capped CdTe quantum dots of different sizes on the HSA was investigated. After correction for the inner filter effect, the fluorescence emission spectra and synchronous fluorescence spectra showed that the microenvironment of aromatic acid residues in the protein was slightly changed when the glutathione (GSH)-cadmium telluride (CdTe) QDs was added, and GSH-CdTe QDs with larger particle size exhibited a much higher effect on HSA than the small particles. Although a ground-state complex between HSA and GSH-CdTe QDs was formed, the UV-vis absorption and circular dichroism spectroscopic results did not find appreciable conformational changes of HSA. ITC has been used for the first time to characterize the binding of QDs with HSA. The ITC results revealed that the binding was a thermodynamically spontaneous process mainly driven by hydrophobic interactions, and the binding constant tended to increase as the GSH-CdTe QDs size increased. These findings are helpful in understanding the bioactivities of QDs in vivo and can be used to assist in the design of biocompatible and stable QDs.