The increasing frequency and magnitude of viral outbreaks in recent decades, epitomized by the COVID-19 pandemic, has resulted in an urgent need for rapid and sensitive diagnostic methods. Here, we present a methodology for virus detection and identification that uses a convolutional neural network to distinguish between microscopy images of fluorescently labeled intact particles of different viruses. Our assay achieves labeling, imaging, and virus identification in less than 5 min and does not require any lysis, purification, or amplification steps. The trained neural network was able to differentiate SARS-CoV-2 from negative clinical samples, as well as from other common respiratory pathogens such as influenza and seasonal human coronaviruses. We were also able to differentiate closely related strains of influenza, as well as SARS-CoV-2 variants. Additional and novel pathogens can easily be incorporated into the test through software updates, offering the potential to rapidly utilize the technology in future infectious disease outbreaks or pandemics. Single-particle imaging combined with deep learning therefore offers a promising alternative to traditional viral diagnostic and genomic sequencing methods and has the potential for significant impact.