The regulation of gene expression programmes is essential for the generation of diverse cell types during development and for adaptation to environmental signals. RNA polymerase II (RNAPII) transcribes genetic information and coordinates the recruitment of accessory proteins that are responsible for the establishment of active chromatin states and transcript maturation. RNAPII is post-translationally modified at active genes during transcription initiation, elongation and termination, and thereby recruits specific histone and RNA modifiers. RNAPII complexes are also located at silent genes in promoter-proximal paused configurations that provide dynamic transcriptional regulation downstream from initiation. In embryonic stem cells, silent developmental regulator genes that are repressed by Polycomb are associated with a form of RNAPII that can elongate through coding regions but that lacks the post-translational modifications that are important for coupling RNA synthesis to co-transcriptional maturation. Here, we discuss the mechanisms through which the transcription of silent genes might be dissociated from productive expression, and the sophisticated interplay between the transcriptional machinery, Polycomb repression and RNA processing.