Normal breast luminal epithelial progenitors have been implicated as cell of origin in basal-like breast cancer, but their anatomical localization remains understudied. Here, we combine collection under the microscope of organoids from reduction mammoplasties and single-cell mRNA sequencing (scRNA-seq) of FACS-sorted luminal epithelial cells with multicolor imaging to profile ducts and terminal duct lobular units (TDLUs) and compare them with breast cancer subtypes. Unsupervised clustering reveals eleven distinct clusters and a differentiation trajectory starting with keratin 15 (K15) progenitors enriched in ducts. Spatial mapping of luminal progenitors is confirmed at the protein level by staining with critical duct markers. Comparison of the gene expression profiles of normal luminal cells with those of breast cancer subtypes suggests a strong correlation between normal breast ductal progenitors and basal-like breast cancer. We propose that K15 basal-like breast cancers originate in ductal progenitors, which emphasizes the importance of not only lineages but also cellular position within the ductal-lobular tree.