Neurotransmission is based on the exocytic fusion of synaptic vesicles (SVs) followed by endocytic membrane retrieval and the reformation of SVs. Recent data suggest that at physiological temperature SVs are internalized via clathrin-independent ultrafast endocytosis (UFE) within hundreds of milliseconds, while other studies have postulated a key role for clathrin-mediated endocytosis (CME) of SV proteins on a timescale of seconds to tens of seconds. Here we demonstrate using cultured hippocampal neurons as a model that at physiological temperature SV endocytosis occurs on several timescales from less than a second to several seconds, yet, is largely independent of clathrin. Clathrin-independent endocytosis (CIE) of SV membranes is mediated by actin-nucleating formins such as mDia1, which are required for the formation of presynaptic endosome-like vacuoles from which SVs reform. Our results resolve previous discrepancies in the field and suggest that SV membranes are predominantly retrieved via CIE mediated by formin-dependent actin assembly.