The cellular plasticity of pluripotent stem cells is thought to be sustained by genomic regions that display both active and repressive chromatin properties. These regions exhibit low levels of gene expression, yet the mechanisms controlling these levels remain unknown. Here, we describe Elongin BC as a binding factor at the promoters of bivalent sites. Biochemical and genome-wide analyses show that Elongin BC is associated with Polycomb Repressive Complex 2 (PRC2) in pluripotent stem cells. Elongin BC is recruited to chromatin by the PRC2-associated factor EPOP (Elongin BC and Polycomb Repressive Complex 2 Associated Protein, also termed C17orf96, esPRC2p48, E130012A19Rik), a protein expressed in the inner cell mass of the mouse blastocyst. Both EPOP and Elongin BC are required to maintain low levels of expression at PRC2 genomic targets. Our results indicate that keeping the balance between activating and repressive cues is a more general feature of chromatin in pluripotent stem cells than previously appreciated.