The inferior colliculus (IC) is a central hub in the ascending auditory brainstem. It hosts many neurons tuned to interaural time differences (ITDs). ITD tuning, however, is already observed and generated one synapse upstream in the superior olivary complex and the physiological mechanisms as well as the functional purpose of the IC projection remain partially unresolved. Here, we argue that combining ITD sensitive inputs from medial superior olive (MSO) and lateral superior olive (LSO) requires a temporally well adjusted delay of cross-hemispheric fibers from LSO to IC, given the fast synaptic kinetics of IC neurons. We present a normative model of the midbrain auditory circuitry that finds an optimal cross-hemispheric delay of 0.3 cycles and optimal synaptic strengths by maximizing the firing rate of IC neurons for a stimulus at a given ITD. The model suggests that, by varying the relative synaptic weight of MSO and LSO input, individual neurons are optimized to transmit information of all sound sources in a complex auditory scene. ITD tuning of IC neurons would then results as a side effect. The model focuses on the low-frequency range, is consistent with the distribution of best ITDs observed in experimental recordings and performs close to optimal in sound source reconstruction.