Amino acids signal to the mTOR complex I (mTORC1) growth pathway through the Rag GTPases. Multiple distinct complexes regulate the Rags, including GATOR1, a GTPase activating protein (GAP), and GATOR2, a positive regulator of unknown molecular function. Arginine stimulation of cells activates mTORC1, but how it is sensed is not well understood. Recently, SLC38A9 was identified as a putative lysosomal arginine sensor required for arginine to activate mTORC1 but how arginine deprivation represses mTORC1 is unknown. Here, we show that CASTOR1, a previously uncharacterized protein, interacts with GATOR2 and is required for arginine deprivation to inhibit mTORC1. CASTOR1 homodimerizes and can also heterodimerize with the related protein, CASTOR2. Arginine disrupts the CASTOR1-GATOR2 complex by binding to CASTOR1 with a dissociation constant of ~30 μM, and its arginine-binding capacity is required for arginine to activate mTORC1 in cells. Collectively, these results establish CASTOR1 as an arginine sensor for the mTORC1 pathway.