Embodied emissions from the production of building materials account for 17% of China's carbon dioxide (CO) emissions and are important to focus on as China aims to achieve its carbon neutrality goals. However, there is a lack of systematic assessments on embodied emissions reduction potential of building materials that consider both the heterogeneous industrial characteristics as well as the Chinese buildings sector context. Here, we developed an integrated model that combines future demand of building materials in China with the strategies to reduce CO emissions associated with their production, using, and recycling. We found that measures to improve material efficiency in the value-chain has the largest CO mitigation potential before 2030 in both Low Carbon and Carbon Neutrality Scenarios, and continues to be significant through 2060. Policies to accelerate material efficiency practices, such as incorporating embodied emissions in building codes and conducting robust research, development, and demonstration (RD&D) in carbon removal are critical.