The data on the outcome of breast invasive lobular carcinoma (ILC) are conflicting. In addition, the prognostic effect of molecular subtypes on ILC remains unclear. In this study, the clinicopathological and prognostic data between 269 ILC and 816 invasive ductal carcinoma (IDC) cases in a Chinese population were extensively compared, with a median follow-up time of 7.8 years. Compared with the IDC group, ILC tumors had more lymph node invasion, hormonal receptor positivity, and human epidermal growth factor receptor 2 (HER2) negativity. ILC patients showed overall survival (OS) and recurrence/metastasis-free survival (RFS) rates similar to those of IDC patients but exhibited worse disease-free survival (DFS) rate because of the higher rate of contralateral breast cancer (BC). Further analysis showed that OS, RFS, and DFS were similar between ILC and IDC patients in the subgroups of luminal A and triple-negative BC with HER2 negativity but were worse in ILC patients than those in IDC patients in the subgroups of luminal B and HER2 overexpression with positive HER2 expression. Multivariate analysis indicated HER2 positivity as an independent risk factor for OS, RFS, and DFS of ILC patients, which increased the risk in the ILC group than that in IDC group. The interaction of HER2 and ILC was also defined as an independent risk factor for OS, RFS, and DFS of the entire population. In conclusion, overexpression of HER2 exhibited stronger negative effect on the prognosis of ILC patients than that in IDC patients, suggesting that treatment targeting HER2 is crucial for this BC subgroup.