Biomolecular and physical cues of the extracellular matrix environment regulate collective cell dynamics and tissue patterning. Nonetheless, how the viscoelastic properties of the matrix regulate collective cell spatial and temporal organization is not fully understood. Here we show that the passive viscoelastic properties of the matrix encapsulating a spheroidal tissue of breast epithelial cells guide tissue proliferation in space and in time. Matrix viscoelasticity prompts symmetry breaking of the spheroid, leading to the formation of invading finger-like protrusions, YAP nuclear translocation and epithelial-to-mesenchymal transition both in vitro and in vivo in a Arp2/3-complex-dependent manner. Computational modelling of these observations allows us to establish a phase diagram relating morphological stability with matrix viscoelasticity, tissue viscosity, cell motility and cell division rate, which is experimentally validated by biochemical assays and in vitro experiments with an intestinal organoid. Altogether, this work highlights the role of stress relaxation mechanisms in tissue growth dynamics, a fundamental process in morphogenesis and oncogenesis.