Considering the high consumption of coffee in Brazil, this study aimed to investigate the relationship between coffee consumption and the intake of added sugar, non-caloric sweeteners, sugary beverages, and foods. A modified case-crossover study was conducted using data from the national "Household Budget Survey (POF)" which 38,854 participants. Dietary intake was assessed using 24-h recalls on two non-consecutive days. Days with and without coffee consumption were compared (case-crossover) to evaluating the effects on sugar, sweeteners, sugary foods, sugar-sweetened beverages (SSB), and milk. 87% of Brazilians aged 10 and older consumed coffee on a giving day. The consumption of all food groups decreased as days of coffee consumption increased, except for non-caloric sweeteners and sugar, which increased. In the case-crossover analysis (2192 men and 2580 women), women who consumed coffee on one of the two days showed an increase of 10 g of sugar and of 0.10 mL (2 drops) of sweeteners. For men values were 8 g and 0.05 mL. Also, women reduced sugar-sweetened beverages (-56.8 mL/day), while men reduced milk intake (-25.9 mL/day). Coffee consumption was associated with increased intake of sugar and non-caloric sweeteners and lower intake of sweets, SSB, and milk. Reducing sweets and SSB is beneficial but increasing sweeteners and reducing milk are not. Potential strategies include encouraging the use of milk in coffee instead of sugar and sweeteners, as well as reducing the size of sugar sachets, which in Brazil typically range from 5 g to 8 g.