CCCTC-binding factor (CTCF) and cohesin are key players in three-dimensional chromatin organization. The topologically associating domains (TADs) demarcated by CTCF are remarkably well conserved between species, although genome-wide CTCF binding has diverged substantially following transposon-mediated motif expansions. Therefore, the CTCF consensus motif poorly predicts TADs, and additional factors must modulate CTCF binding and subsequent TAD formation. Here, we demonstrate that the ChAHP complex (CHD4, ADNP, HP1) competes with CTCF for a common set of binding motifs. In Adnp knockout cells, novel insulated regions are formed at sites normally bound by ChAHP, whereas proximal canonical boundaries are weakened. These data reveal that CTCF-mediated loop formation is modulated by a distinct zinc-finger protein complex. Strikingly, ChAHP-bound loci are mainly situated within less diverged SINE B2 transposable elements. This implicates ChAHP in maintenance of evolutionarily conserved spatial chromatin organization by buffering novel CTCF binding sites that emerged through SINE expansions.