A fundamental question linking population genetics and community ecology is how adaptive processes (e.g., natural selection) and neutral processes (e.g., drift-migration equilibrium) underpin the species-genetic diversity correlation (SGDC). Here, we combine genome scans and outlier loci detection with community analysis to separately test for neutral and nonneutral SGDCs in four species of stream insect. We sampled 60 localities in Japan and examined the relationships among population AFLP band richness (Br), taxon richness of the total community (S) and of the trophic guild (S ), and 15 habitat parameters that could potentially drive adaptation and influence richness. Neutral Br was positively correlated with S only in the dominant species of these communities, suggesting Br may be constrained when intraspecific competition is pronounced. Nonneutral Br was correlated with S in a species restricted to high elevations where habitat heterogeneity was highest. Community distance and genetic distance (β-SGDC) was correlated in two of the four species at both neutral and nonneutral loci. Distance-based redundancy analysis found geographic isolation and elevation to drive divergence of both communities and populations. This suggests that both neutral and adaptive divergence occurred through the shared influences of geographic isolation and local adaptation at the two levels of diversity.