Congenital aphantasia is a recently characterized variation of experience defined by the inability to form voluntary visual imagery, in individuals who are otherwise high performing. Because of this specific deficit to visual imagery, individuals with aphantasia serve as an ideal group for probing the nature of representations in visual memory, particularly the interplay of object, spatial, and symbolic information. Here, we conducted a large-scale online study of aphantasia and revealed a dissociation in object and spatial content in their memory representations. Sixty-one individuals with aphantasia and matched controls with typical imagery studied real-world scene images, and were asked to draw them from memory, and then later copy them during a matched perceptual condition. Drawings were objectively quantified by 2,795 online scorers for object and spatial details. Aphantasic participants recalled significantly fewer objects than controls, with less color in their drawings, and an increased reliance on verbal scaffolding. However, aphantasic participants showed high spatial accuracy equivalent to controls, and made significantly fewer memory errors. These differences between groups only manifested during recall, with no differences between groups during the matched perceptual condition. This object-specific memory impairment in individuals with aphantasia provides evidence for separate systems in memory that support object versus spatial information. The study also provides an important experimental validation for the existence of aphantasia as a variation in human imagery experience.