Breast Cancer (BC) is the most commonly diagnosed cancer and second leading cause of mortality among women. About 1 in 8 US women (about 13%) will develop invasive BC throughout their lifetime. Early detection of this life-threatening disease not only increases the survival rate but also reduces the treatment cost. Fortunately, advancements in radiographic imaging like "Mammograms", "Computed Tomography (CT)", "Magnetic Resonance Imaging (MRI)", "3D Mammography", and "Histopathological Imaging (HI)" have made it feasible to diagnose this life-taking disease at an early stage. However, the analysis of radiographic images and Histopathological images is done by experienced radiologists and pathologists, respectively. The process is not only costly but also error-prone. Over the last ten years, Computer Vision and Machine Learning (ML) have transformed the world in every way possible. Deep learning (DL), a subfield of ML has shown outstanding results in a variety of fields, particularly in the biomedical industry, because of its ability to handle large amounts of data. DL techniques automatically extract the features by analyzing the high dimensional and correlated data efficiently. The potential and ability of DL models have also been utilized and evaluated in the identification and prognosis of BC, utilizing radiographic and Histopathological images, and have performed admirably. However, AI has shown good claims in retrospective studies only. External validations are needed for translating these cutting-edge AI tools as a clinical decision maker. The main aim of this research work is to present the critical analysis of the research and findings already done to detect and classify BC using various imaging modalities including "Mammography", "Histopathology", "Ultrasound", "PET/CT", "MRI", and "Thermography". At first, a detailed review of the past research papers using Machine Learning, Deep Learning and Deep Reinforcement Learning for BC classification and detection is carried out. We also review the publicly available datasets for the above-mentioned imaging modalities to make future research more accessible. Finally, a critical discussion section has been included to elaborate open research difficulties and prospects for future study in this emerging area, demonstrating the limitations of Deep Learning approaches.