Polycomb bodies are foci of Polycomb proteins in which different Polycomb target genes are thought to co-localize in the nucleus, looping out from their chromosomal context. We have shown previously that insulators, not Polycomb response elements (PREs), mediate associations among Polycomb Group (PcG) targets to form Polycomb bodies. Here we use live imaging and 3C interactions to show that transgenes containing PREs and endogenous PcG-regulated genes are targeted by insulator proteins to different nuclear structures depending on their state of activity. When two genes are repressed, they co-localize in Polycomb bodies. When both are active, they are targeted to transcription factories in a fashion dependent on Trithorax and enhancer specificity as well as the insulator protein CTCF. In the absence of CTCF, assembly of Polycomb bodies is essentially reduced to those representing genomic clusters of Polycomb target genes. The critical role of Trithorax suggests that stable association with a specialized transcription factory underlies the cellular memory of the active state.