Erosion and the stripping effect of moisture on asphalt mixtures is one of the main reasons for the shortened service life of asphalt pavements. The common mean of preventing asphalt pavements from being damaged by moisture is adding anti-stripping agents (ASAs) to asphalt mixtures. However, the effect regularity and mechanism of anti-stripping agents on the physicochemical properties of asphalt is not exactly defined. This study compared the physical properties of ASA-modified asphalt (AMAs) to determine the optimal dosage and investigated the rheological and adhesion properties. Based on the roller bottle method and water immersion method, the moisture susceptibility of AMAs with three particle sizes was investigated. The results showed that the modification of asphalt using anti-stripping agents was a physical modification. At the optimum dosage of anti-stripping agents (0.3%), the basic physical properties of AMA1 were the most desirable. ASA2 increased the resistance of asphalt for deformation at high temperature by 46%, and AMA3 had the best low-temperature performance. ASAs enhanced the dispersed and polar components in the asphalt binder, improving the adhesion energy of asphalt. AMA3 had the strongest adhesion to the aggregate, with an increase in adhesion work by 2.8 times and a 45% of increase in ER value. This was attributed to ASA3 containing with a large number of metal cations and polar functional groups. It was shown that ASAs provided the most improvement in the anti-stripping performance of asphalt mixtures with 9.5-13.2 mm particles. The amide ASA, phosphate ASA and aliphatic amine ASA improved the water damage resistance of asphalt by 65%, 45% and 78%, respectively. This study can help engineers realize the effects of different types of ASAs on the physicochemical properties of asphalt and select the most suitable type of ASAs according to the service requirements.