Vaccinia virus (VACV) was used for smallpox eradication, but its ultimate origin remains unknown. The genetic relationships among vaccine stocks are also poorly understood. We analyzed 63 vaccine strains with different origin, as well horsepox virus (HPXV). Results indicated the genetic diversity of VACV is intermediate between variola and cowpox viruses, and that mutation contributed more than recombination to VACV evolution. STRUCTURE identified 9 contributing subpopulations and showed that the lowest drift was experienced by the ancestry components of Tian Tan and HPXV/Mütter/Mulford genomes. Subpopulations that experienced very strong drift include those that contributed the ancestry of MVA and IHD-W, in good agreement with the very long passage history of these vaccines. Another highly drifted population contributed the full ancestry of viruses sampled from human/cattle infections in Brazil and, partially, to IOC clones, strongly suggesting that the recurrent infections in Brazil derive from the spillback of IOC to the feral state.