The gut microbiota in the human body is an important component that plays a pivotal role in the ability of the host to prevent diseases and recover from these diseases. If the human microbiome changes for any reason, it affects the overall functioning of the host. Healthy and vigorous gut microbiota require dietary fiber supplementation. Recently, oligosaccharides have been found to play a significant role in the modulation of microbiota. Several such oligosaccharides, i.e., xylooligosaccharides (XOS), mannooligosaccharides (MOS), and arabino-xylooligosaccharides (AXOS), are derived from hemicellulosic macromolecules such as xylan, mannan, and arabino-xylan, respectively. These oligosaccharides serve as substrates for the probiotic production of health-promoting substances (short-chain fatty acids, branched chain amino acids etc.), which confer a variety of health benefits, including the prevention of some dreaded diseases. Among hemicellulose-derived oligosaccharides (HDOs), XOS have been largely explored, whereas, studies on MOS and AXOS are currently underway. HDOs, upon ingestion, help reduce morbidities by lowering populations of harmful or pathogenic bacteria. The ATP-binding cassette (ABC) transporters are mainly utilized for the uptake of oligosaccharides in probiotics. Butyrate generated by the selective fermentation of oligosaccharides, along with other short-chain fatty acids, reduces gut inflammation. Overall, oligosaccharides derived from hemicelluloses show a similar potential as conventional prebiotics and can be supplemented as functional foods. This review summarizes the role of HDOs in the alleviation of autoimmune diseases (inflammatory bowel disease, Crohn's disease), diabetes, urinary tract infection, cardiovascular diseases, and antimicrobial resistance (AMR) through the modulation of the gut microbiota. The mechanism of oligosaccharide utilization and disease mitigation is also explained.