Recent advances in microcircuit analysis of nervous systems have revealed a plethora of mutual connections between inhibitory interneurons across many different species and brain regions. The abundance of these mutual connections has not been fully explained. Strikingly, we show that neural circuits with mutually inhibitory connections are able to rapidly and flexibly switch between distinct functions. That is, multiple functions coexist for a single set of synaptic weights. Here, we develop a theoretical framework to explain how inhibitory recurrent circuits give rise to this flexibility and show that mutual inhibition doubles the number of cusp bifurcations in small neural circuits. As a concrete example, we study a class of functional motifs we call coupled recurrent inhibitory and recurrent excitatory loops (CRIRELs). These CRIRELs have the advantage of being both multi-functional and controllable, performing a plethora of functions, including decisions, memory, toggle, and so forth. Finally, we demonstrate how mutual inhibition maximizes storage capacity for larger networks.