Image-based diagnosis has become a crucial tool in the identification and management of various cancers, particularly lung and colon cancer. This review delves into the latest advancements and ongoing challenges in the field, with a focus on deep learning, machine learning, and image processing techniques applied to X-rays, CT scans, and histopathological images. Significant progress has been made in imaging technologies like computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET), which, when combined with machine learning and artificial intelligence (AI) methodologies, have greatly enhanced the accuracy of cancer detection and characterization. These advances have enabled early detection, more precise tumor localization, personalized treatment plans, and overall improved patient outcomes. However, despite these improvements, challenges persist. Variability in image interpretation, the lack of standardized diagnostic protocols, unequal access to advanced imaging technologies, and concerns over data privacy and security within AI-based systems remain major obstacles. Furthermore, integrating imaging data with broader clinical information is crucial to achieving a more comprehensive approach to cancer diagnosis and treatment. This review provides valuable insights into the recent developments and challenges in image-based diagnosis for lung and colon cancers, underscoring both the remarkable progress and the hurdles that still need to be overcome to optimize cancer care.