High-fiber diet interventions have been proven to be beneficial for gut microbiota and glycemic control in diabetes patients. However, the effect of a low level of fiber in habitual diets remains unclear. This study aims to examine the associations of habitual dietary fiber intake with gut microbiome profiles among Chinese diabetes patients and identify differential taxa that mediated associations of dietary fiber with HbA1c level. Two cross-sectional studies and one longitudinal study were designed based on two follow-up surveys in a randomized trial conducted during 2015−2017. The study included 356 and 310 participants in the first and second follow-ups, respectively, with 293 participants in common in both surveys. Dietary fiber intake was calculated based on a 3-day 24-h diet recall at each survey and was classified into a lower or a higher group according to the levels taken based on the two surveys using 7.2 g/day as a cut-off value. HbA1c was assayed to assess glycemic status using a cut-off point of 7.0% and 8.0%. Microbiome was profiled by 16S rRNA sequencing. A high habitual dietary fiber intake was associated with a decrease in α-diversity in both the cross-sectional and longitudinal analyses. At the first follow−up, phylum Firmicutes and Fusobacteria were negatively associated with a higher dietary fiber intake (p < 0.05, Q < 0.15); at the second follow-up, genus Adlercreutzia, Prevotella, Ruminococcus, and Desulfovibrio were less abundant in patients taking higher dietary fiber (p < 0.05, Q < 0.15); genus Desulfovibrio and Ruminococcaceae (Unknown), two identified differential taxa by HbA1c level, were negatively associated with dietary fiber intake in both the cross-sectional and longitudinal analyses, and mediated the dietary fiber-HbA1c associations among patients taking dietary fiber ≥ 7.2 g/day (mediation effect β [95%CI]: −0.019 [−0.043, −0.003], p = 0.018 and −0.019 [−0.046, −0.003], p = 0.016). Our results suggest that habitual dietary fiber intake has a beneficial effect on gut microbiota in Chinese diabetes patients. Further studies are needed to confirm our results.