Persistent homology offers a powerful tool for extracting hidden topological signals from brain networks. It captures the evolution of topological structures across multiple scales, known as filtrations, thereby revealing topological features that persist over these scales. These features are summarized in persistence diagrams, and their dissimilarity is quantified using the Wasserstein distance. However, the Wasserstein distance does not follow a known distribution, posing challenges for the application of existing parametric statistical models. To tackle this issue, we introduce a unified topological inference framework centered on the Wasserstein distance. Our approach has no explicit model and distributional assumptions. The inference is performed in a completely data driven fashion. We apply this method to resting-state functional magnetic resonance images (rs-fMRI) of temporal lobe epilepsy patients collected from two different sites: the University of Wisconsin-Madison and the Medical College of Wisconsin. Importantly, our topological method is robust to variations due to sex and image acquisition, obviating the need to account for these variables as nuisance covariates. We successfully localize the brain regions that contribute the most to topological differences. A MATLAB package used for all analyses in this study is available at https://github.com/laplcebeltrami/PH-STAT.