SETDB1 is an essential histone methyltransferase that deposits histone H3 lysine 9 trimethylation (H3K9me3) to transcriptionally repress genes and repetitive elements. The function of differential H3K9me3 enrichment between cell-types remains unclear. Here, we demonstrate mutual exclusivity of H3K9me3 and CTCF across mouse tissues from different developmental timepoints. We analyze SETDB1 depleted cells and discover that H3K9me3 prevents aberrant CTCF binding independently of DNA methylation and H3K9me2. Such sites are enriched with SINE B2 retrotransposons. Moreover, analysis of higher-order genome architecture reveals that large chromatin structures including topologically associated domains and subnuclear compartments, remain intact in SETDB1 depleted cells. However, chromatin loops and local 3D interactions are disrupted, leading to transcriptional changes by modifying pre-existing chromatin landscapes. Specific genes with altered expression show differential interactions with dysregulated cis-regulatory elements. Collectively, we find that cell-type specific targets of SETDB1 maintain cellular identities by modulating CTCF binding, which shape nuclear architecture and transcriptomic networks.