The presence of large hydrophobic aromatic residues in cell-penetrating peptides or proteins has been demonstrated to be advantageous for their cell penetration. This phenomenon has also been observed when AuNPs were modified with peptides containing aromatic amino acids. However, it is still not clear how the presence of hydrophobic and aromatic groups on the surface of anionic AuNPs affects their interaction with lipid bilayers. Here, we studied the interaction of a range of anionic amphiphilic AuNPs coated by different combinations of hydrophobic and anionic ligands with four different types of synthetic lipid vesicles. Our results demonstrated the important role of the surface aromatic or bulky groups, relative to the hydrocarbon chains, in the interaction of anionic AuNPs with lipid bilayers. Hydrophobic interaction itself arising from the insertion of aromatic/bulky ligands on the surface of AuNPs into lipid bilayers is sufficiently strong to cause overt disruption of lipid vesicles and cell membranes. Moreover, by comparing the results obtained from AuNPs coated with aromatic ligands and cyclohexyl ligands lacking aromaticity respectively, we demonstrated that the bulkiness of the terminal groups in hydrophobic ligands instead of the aromatic character might be more important to the interaction of AuNPs with lipid bilayers. Finally, we further correlated the observation on model liposomes with that on cell membranes, demonstrating that AuNPs that are more disruptive to the more negatively charged liposomes are also substantially more disruptive to cell membranes. In addition, our results revealed that certain cellular membrane domains that are more susceptible to disruption caused by hydrophobic interactions with nanoparticle surfaces might determine the threshold of AuNP-mediated cytotoxicity.