Plant cell walls are essential for growth. The cell wall hemicellulose xyloglucan (XyG) is produced in the Golgi apparatus before secretion. Loss of the Arabidopsis galactosyltransferase MURUS3 (MUR3) decreases XyG d-galactose side chains and causes intracellular aggregations and dwarfism. It is unknown how changing XyG synthesis can broadly impact organelle organization and growth. We show that intracellular aggregations are not unique to mur3 and are found in multiple mutant lines with reduced XyG D-galactose side chains. mur3 aggregations disrupt subcellular trafficking and induce formation of intracellular cell-wall-like fragments. Addition of d-galacturonic acid onto XyG can restore growth and prevent mur3 aggregations. These results indicate that the presence, but not the composition, of XyG side chains is essential, likely by ensuring XyG solubility. Our results suggest that XyG polysaccharides are synthesized in a highly substituted form for efficient secretion and then later modified by cell-wall-localized enzymes to fine-tune cell wall properties.