Activation-induced marker (AIM) assays identify antigen (Ag)-specific T cells, but recent studies revealed AIM T helper cell 17 (T17)-like (CCR6) and circulating T follicular helper cells (cTfh) were not associated with peptide/HLA tetramer staining. We show that CD39 regulatory T cell (T)-like and CD26 T22-like cells undergo T cell receptor (TCR)-independent activation by cytokines during Ag stimulation, leading to nonspecific up-regulation of AIM readouts. Transcriptional analysis enabled discrimination of bona fide Ag-specific T cells from cytokine-activated T and T22 cells. CXCR4 down-regulation emerged as a hallmark of clonotypic expansion and TCR-dependent activation in memory CD4 T cells and cTfh. By tracking tetramer-binding cells upon Ag restimulation, we demonstrated that CXCR4-CD137 cells provided a more accurate measure of Ag-specificity than standard AIM readouts. This modified assay excluded the predominantly CCR6 cytokine-activated T cells that contributed to an average 12-fold overestimation of the Ag-specific population. Our findings provide an accurate approach to characterize genuine Ag-specific T cells.