In cases of brain disease such as temporal lobe epilepsy (TLE), damage may lead to functional reorganization and a shift in language dominance to homolog regions in the other hemisphere. If the effects of TLE on language dominance are hemisphere-focused, then brain regions and connections involved in word reading should be less left-lateralized in left temporal lobe epilepsy (lTLE) than right temporal lobe epilepsy (rTLE) or healthy controls, and the opposite effect should be observed in patients with rTLE. In our study, functional magnetic resonance imaging (fMRI) showed that patients with rTLE had more strongly lateralized left hemisphere (LH) activation than patients with lTLE and healthy controls in language-related brain regions (pars opercularis and fusiform gyrus (FuG)). Corresponding with this difference, diffusion tensor imaging (DTI) found differences in connectivity indicative of patients with lTLE having greater tract integrity than patients with rTLE in the right hemisphere (RH) uncinate fasciculus (UF), inferior longitudinal fasciculus (ILF), and inferior fronto-occipital fasciculus (IFOF) using the network-based statistic analysis method. The UF, ILF, and IFOF tract integrity have previously been associated with lexical (whole-word) processing abilities. Multivariate distance matrix regression provided converging evidence for regions of the IFOF having different connectivity patterns between groups with lTLE and rTLE. This research demonstrates language lateralization differences between patient groups with lTLE and rTLE, and corresponding differences in the connectivity strength of the ILF, IFOF, and UF. This research provides a novel approach to measuring lateralization of language in general, and the fMRI and DTI findings were integral for guiding the neurosurgeons performing the TLE resections. This approach should inform future studies of language lateralization and language reorganization in patients such as those with TLE.