INTRODUCTION: Positron emission tomography (PET) imaging targeting neurofibrillary tau tangles is increasingly used in the study of Alzheimer's disease (AD), but its utility may be limited by conventional quantitative or qualitative evaluation techniques in earlier disease states. Convolutional neural networks (CNNs) are effective in learning spatial patterns for image classification. METHODS: 18F-MK6240 (n = 320) and AV-1451 (n = 446) PET images were pooled from multiple studies. We performed iterations with differing permutations of radioligands, heuristics, and architectures. Performance was compared to a standard region of interest (ROI)-based approach on prediction of memory impairment. We visualized attention of the network to illustrate decision making. RESULTS: Overall, models had high accuracy (> 80%) with good average sensitivity and specificity (75% and 82%, respectively), and had comparable or higher accuracy to the ROI standard. Visualizations of model attention highlight known characteristics of tau radioligand binding. DISCUSSION: CNNs could improve tau PET's role in early disease and extend the utility of tau PET across generations of radioligands.