VISTA is a key immune checkpoint receptor under investigation as a target for cancer immunotherapy. However, a better understanding of the signaling mechanisms of VISTA is needed to optimize the therapeutic potential. Here, we identified a conserved four amino acid (NPGF) intracellular motif in VISTA that suppresses cell proliferation by constraining cell-intrinsic growth receptor signaling. A class of triple-negative breast cancers (TNBC) with high VISTA expression and low proliferative index was identified and characterized. The NPGF motif bound to the adapter protein NUMB and recruited Rab11 endosomal recycling machinery. The NPGF motif sequestered NUMB at endosomes, which interfered with EGFR trafficking and signaling to suppress tumor growth. These tumor suppressive effects did not require canonical VISTA ligands or a functioning immune system. Mutation of the VISTA NPGF domain reverted VISTA-induced growth suppression in multiple breast cancer mouse models. The NPGF motif was also required for response of VISTA+ TNBCs to VISTA-blocking antibodies. These results define a mechanism by which VISTA recruits adapter proteins to control malignant epithelial cell growth and signaling. They also define distinct intracellular residues that are critical for response to therapeutic antibodies that could be exploited to improve immunotherapy.