Surface-Enhanced Raman Spectroscopy (SERS) is a powerful analytical technique for the detection of small analytes with great potential for medical diagnostic applications. Its high sensitivity and excellent molecular specificity, which stems from the unique fingerprint of molecular species, have been applied toward the detection of different types of cancer. The noninvasive and rapid detection offered by SERS highlights its applicability for point-of-care (PoC) deployment for cancer diagnosis, screening, and staging, as well as for predicting tumor recurrence and treatment monitoring. This review provides an overview of the progress in label-free (direct) SERS-based chemical detection for cancer diagnosis with the main focus on the advances in the design and preparation of SERS substrates on the basis of metal nanoparticle structures formed via bottom-up strategies. It begins by introducing a synopsis of the working principles of SERS, including key chemometric approaches for spectroscopic data analysis. Then it introduces the advances of label-free sensing with SERS in cancer diagnosis using biofluids (blood, urine, saliva, sweat) and breath as the detection media. In the end, an outlook of the advances and challenges in cancer diagnosis via SERS is provided.