Enzymes play a vital role in life processes; they control chemical reactions and allow functional cycles to be synchronized. Many enzymes harness large-scale motions of their domains to achieve tremendous catalytic prowess and high selectivity for specific substrates. One outstanding example is provided by the three-domain enzyme adenylate kinase (AK), which catalyzes phosphotransfer between ATP to AMP. Here we study the phenomenon of substrate inhibition by AMP and its correlation with domain motions. Using single-molecule FRET spectroscopy, we show that AMP does not block access to the ATP binding site, neither by competitive binding to the ATP cognate site nor by directly closing the LID domain. Instead, inhibitory concentrations of AMP lead to a faster and more cooperative domain closure by ATP, leading in turn to an increased population of the closed state. The effect of AMP binding can be modulated through mutations throughout the structure of the enzyme, as shown by the screening of an extensive AK mutant library. The mutation of multiple conserved residues reduces substrate inhibition, suggesting that substrate inhibition is an evolutionary well conserved feature in AK. Combining these insights, we developed a model that explains the complex activity of AK, particularly substrate inhibition, based on the experimentally observed opening and closing rates. Notably, the model indicates that the catalytic power is affected by the microsecond balance between the open and closed states of the enzyme. Our findings highlight the crucial role of protein motions in enzymatic activity.