: Transitions toward more sustainable food systems may become rather polarized, particularly in the plant-based vs. animal-based debate. These discussions, however, are often based on environmental impact data from individual products or product groups and do not consider that the products together should form a nutrient-adequate diet that is also affordable. Linear programming approaches are ideal for exploring the impact of changes in diet composition on environmental impact and price while ensuring nutrient adequacy. : In this study, we used a linear programming system, Optimeal 2.0, based on the diet in the Netherlands to explore the impact of changing the contributions of specific food groups on the environmental impact and price of the diet. : Reducing the amount of meat products in the diet reduced the environmental impact but increased the price. On the other hand, when dairy products were reduced or even omitted, the environmental impact of the nutrient-adequate optimized diet did not change notably, but prices increased notably. This could be attributed to the fact that the products required to compensate for the nutrient gaps by removing dairy products have equal environmental impact and higher prices. Increasing vegetable or fruit consumption increased price but did not affect the environmental impact nor did it increase the consumption of beans and pulses when kept within realistic levels. : Overall, this work strongly highlights that consideration of 'sustainability' at a food product level is insufficient and that their role in nutrient-adequate diets needs to be considered. Furthermore, dietary shifts should be considered from the perspective of affordability and consumer preferences.