We present a simple method to control the volume change of thermally responsive hydrogel scaffolds, providing a remarkably fast swelling and deswelling response to temperature changes. These scaffolds have 3-dimensional colloidal-network structures which are made from microgel particles while they are above their deswelling transition temperatures. By tuning the cross-link density of the microgel particles, we achieve controllable changes of the volume of the scaffolds in response to temperature. Their fast response rate is determined by the length scale of the unit microgel particles and is not influenced by the properties of the network. The release profile of a model drug (Rhapontin) loaded within the scaffolds can also be regulated by the cross-linking density of the microgel particles. These results offer a new way of fabricating hydrogel scaffolds with tunable matrix geometry and function by adjusting the properties of the unit microgel colloids, without loss of their fast response to temperature change.