Although angiotensin II receptor blockers (ARBs) are more effective in women for either reduction of blood pressure or heart failure (HF), the gender disparities and the impact of class/drug effects on ARBs in relation to cardiac hypertrophy and HF remain unclear. We aimed to investigate the sex-based and drug-specific differences in left ventricular (LV) mass reduction with ARBs. We employed the cohort of 193 hypertensive patients with HF and an LV ejection fraction of ≥ 45% treated with azilsartan or candesartan once daily for 48 weeks as a sub-analysis of the J-TASTE trial. After exclusion of patients without LV mass data nor the drugs, 170 patients were finally enrolled (azilsartan: male, n = 43, female, n = 39 and candesartan: male, n = 52; female, n = 36). We investigated the sex-based differences of the primary endpoint of the change in LV mass as assessed by echocardiography from baseline to the end of the study (48 weeks), and the secondary endpoint of the incidence of the composite cardiovascular endpoint (death from cardiovascular disease or hospitalization for heart failure). In the male stratum, the ratio of patients with > 10% LV mass reduction at 48 weeks was higher in the azilsartan group than candesartan group (40 vs. 19%, p = 0.029). There was no significant difference in LV mass reduction between two groups in the female stratum. There were no differences of the onset of the secondary endpoints between male and female groups, and azilsartan and candesartan groups. The event-free survival rate of the composite cardiovascular endpoints tended to be lower in patients with ≤ 10% than > 10% LV mass reduction (95.3 vs. 100% at 48 weeks, log-rank p = 0.11). In patients with HF, the effectiveness of either azilsartan or candesartan in achieving > 10% LV mass reduction depends on sex. Male is more sensitive to azilsartan than candesartan to achieve cardiac hypertrophy in HF patients.