Hydration water determines the stability and function of phospholipid membranes as well as the interaction of membranes with other molecules. Experiments and simulations have shown that water dynamics slows down dramatically as the hydration decreases, suggesting that the interfacial water that dominates the average dynamics at low hydration is slower than water away from the membrane. Here, based on all-atom molecular dynamics simulations, we provide an interpretation of the slowdown of interfacial water in terms of the structure and dynamics of water-water and water-lipid hydrogen bonds (HBs). We calculate the rotational and translational slowdown of the dynamics of water confined in stacked phospholipid membranes at different levels of hydration, from completely hydrated to poorly hydrated membranes. For all hydrations, we analyze the distribution of HBs and find that water-lipids HBs last longer than water-water HBs and that at low hydration most of the water is in the interior of the membrane. We also show that water-water HBs become more persistent as the hydration is lowered. We attribute this effect (i) to HBs between water molecules that form, in turn, persistent HBs with lipids; (ii) to the hindering of the H-bonding switching between water molecules due to the lower water density at the interface; and (iii) to the higher probability of water-lipid HBs as the hydration decreases. Our interpretation of the large dynamic slowdown in water under dehydration is potentially relevant in understanding membrane biophysics at different hydration levels.