Some oligosaccharides are regarded as biological constituents with benefits to human health in an indirect way. They enter the intestinal tract to be fermented by the gut microbiota, causing changes in the abundance and composition of the gut microbiota and producing fermentation products such as short-chain fatty acids (SCFAs). In this review, the structural features and biological activities of eight common natural oligosaccharides were summarized, including human milk oligosaccharides (HMOS), xylo-oligosaccharides (XOS), arabinoxylo-oligosaccharides (AXOS), isomaltooligosaccharides (IMOS), chitin oligosaccharides (NACOS), mannan-oligosaccharides (MOS), galacto-oligosaccharides (GOS) and fructo-oligosaccharides (FOS). Furthermore, XOS were selected to explain the anti-tumor mechanism mediated by gut microbiota. The review aims to reveal primary structural features of natural functional oligosaccharides related to the biological activities and also provide an explanation of the anti-tumor activity of functional oligosaccharides mediated by the gut microbiota.