During cancer development and progression massive alterations in gene expression are observed. Gene regulation occurs within the context of the 3D genome. However, the impact of disease progression on 3D genome organization remains poorly understood. Using breast cancer as a model, we have profiled the 3D genome throughout the natural course of the disease; from development to progression. Uniquely, we analyzed tumors from the same patients, enabling us to gauge the extent of changes that happen upon metastasis. Our results show that the organization of the genome at the level of topologically associating domains (TADs) and compartments upon tumorigenesis and metastasis, is remarkably stable. However, in pleural metastases, representing heavily pretreated progressive disease, the 3D genome is massively affected, and highly heterogeneous between patients, both on the compartment and TAD level. Our data reveal that disease progression in breast cancer is associated with a progressive unraveling of the 3D genome.