One of the major challenges in the diagnosis of Alzheimer's disease (AD) is to increase the specificity of the early diagnosis. While episodic memory impairment is a sensitive AD marker, other measures are needed to improve diagnostic specificity. A promising biomarker might be a cerebral atrophy of the central olfactory processing areas in the early stages of the disease since an impairment of olfactory identification is present at the clinical stage of AD. Our goal was therefore, (1) to evaluate the grey matter volume (GMV) of central olfactory processing regions in prodromal AD and (2) to assess its association with episodic memory. We included 34 cognitively normal healthy controls (HC), 92 individuals with subjective cognitive decline (SCD), and 40 with mild cognitive impairment (MCI). We performed regions of interest analysis (ROI) using two different approaches, allowing to extract GMV from (1) atlas-based anatomical ROIs and from (2) functional and non-functional subregions of these ROIs (olfactory ROIs and non-olfactory ROIs). Participants with MCI exhibited smaller olfactory ROIs GMV, including significant reductions in the piriform cortex, amygdala, entorhinal cortex, and left hippocampus compared to other groups (p ≤ 0.05, corrected). No significant effect was found regarding anatomical or non-olfactory ROIs GMV. The left hippocampus olfactory ROI GMV was correlated with episodic memory performance (p < 0.05 corrected). Limbic/medial-temporal olfactory processing areas are specifically atrophied at the MCI stage, and the degree of atrophy might predict cognitive decline in AD early stages.