Increasing data has confirmed the potential anticancer properties of Dendrobium, a traditional Chinese herb. However, most anticancer compositions from the plant of Dendrobium were usually extracted by high polar solvent, while weak polar compositions with excellent anticancer activity remained largely unexplored. In this study, the differences between ether extract and ethanol extract of Dendrobium nobile Lindl. on chemical components and anticancer activities were investigated, as well as the anticancer mechanisms among different extracts. The results demonstrated that the ether extract exhibited a stronger anticancer effect than ethanol extract, and its anticancer effect was mainly due to weak polar compounds rather than polysaccharides and alkaloids. Quantitative proteomics suggested that the ether extract significantly stimulated the over-expression of immature proteins, the endoplasmic reticulum stress and unfolded protein response were subsequently induced, the intracellular reactive oxygen species level was seriously elevated, and oxidative stress occurred in the meanwhile. Eventually, autophagy and apoptosis were activated to cause cell death. Our findings demonstrate that the ether extract of D. nobile is a potential candidate for anticancer drug development, and that future research on anticancer drugs derived from medicinal plants should also concentrate on weak polar compounds.