For all sensory organs, the establishment of spatial and temporal cortical resolution is assumed to be initiated by the first sensory experience and a BDNF-dependent increase in intracortical inhibition. To address the potential of cortical BDNF for sound processing, we used mice with a conditional deletion of BDNF in which Cre expression was under the control of the Pax2 or TrkC promoter. BDNF deletion profiles between these mice differ in the organ of Corti (BDNF (Pax2) -KO) versus the auditory cortex and hippocampus (BDNF (TrkC) -KO). We demonstrate that BDNF (Pax2) -KO but not BDNF (TrkC) -KO mice exhibit reduced sound-evoked suprathreshold ABR waves at the level of the auditory nerve (wave I) and inferior colliculus (IC) (wave IV), indicating that BDNF in lower brain regions but not in the auditory cortex improves sound sensitivity during hearing onset. Extracellular recording of IC neurons of BDNF (Pax2) mutant mice revealed that the reduced sensitivity of auditory fibers in these mice went hand in hand with elevated thresholds, reduced dynamic range, prolonged latency, and increased inhibitory strength in IC neurons. Reduced parvalbumin-positive contacts were found in the ascending auditory circuit, including the auditory cortex and hippocampus of BDNF (Pax2) -KO, but not of BDNF (TrkC) -KO mice. Also, BDNF (Pax2) -WT but not BDNF (Pax2) -KO mice did lose basal inhibitory strength in IC neurons after acoustic trauma. These findings suggest that BDNF in the lower parts of the auditory system drives auditory fidelity along the entire ascending pathway up to the cortex by increasing inhibitory strength in behaviorally relevant frequency regions. Fidelity and inhibitory strength can be lost following auditory nerve injury leading to diminished sensory outcome and increased central noise.