The mammalian target of rapamycin complex 1 (mTORC1) integrates nutrients, growth factors, stress, and energy status to regulate cell growth and metabolism. Amino acids promote mTORC1 lysosomal localization and subsequent activation. However, the subcellular location or interacting proteins of mTORC1 under amino acid‐deficient conditions is not completely understood. Here, we identify ADP‐ribosylation factor GTPase‐activating protein 1 (ArfGAP1) as a crucial regulator of mTORC1. ArfGAP1 interacts with mTORC1 in the absence of amino acids and inhibits mTORC1 lysosomal localization and activation. Mechanistically, the membrane curvature‐sensing amphipathic lipid packing sensor (ALPS) motifs that bind to vesicle membranes are crucial for ArfGAP1 to interact with and regulate mTORC1 activity. Importantly, ArfGAP1 represses cell growth through mTORC1 and is an independent prognostic factor for the overall survival of pancreatic cancer patients. Our study identifies ArfGAP1 as a critical regulator of mTORC1 that functions by preventing the lysosomal transport and activation of mTORC1, with potential for cancer therapeutics.