The abundant and renewable resources from fruit by-products are getting emphasis on their valorization. These by-products may contain toxic substances due to factors such as cultivation, harvesting, transportation, preservation, or processing. Hence, presenting scientific overviews of the toxicological qualities and detoxification trends of these by-products are critical for implicating their possible valorization. The present demand for valorization of fruit by-products requires emphasis and methodologies for the detoxification of any toxicants to develop healthier products. This review emphasized the toxicological qualities of by-products from fruits for which the maximum global production occurred in 2022. In this review, heavy metals (arsenic, cadmium, cobalt, chromium, nickel, lead, and mercury), mycotoxins, toxicant organic compounds, anti-nutritional factors, and pesticide/fungicide residues of the selected fruit byproducts were discussed. Current trends to reduce possible toxicants of these by-products during their valorization were emphasized. Novel functional foods valorized from these fruit by-products and future perspectives of detoxification were also focused on in this review.