Low emission zones (LEZs) aiming at improving the air quality in urban areas have been implemented in many European cities. However, studies are limited in evaluating the effects of LEZ, and most of which used simple methods. In this study, a general additive mixed model was utilized to account for confounders in the atmosphere and validate the effects of LEZ on PM and NO concentrations in two German cities. In addition, the effects of LEZ on elemental carbon (EC) and total carbon (TC) in Berlin were also evaluated. The LEZ effects were estimated after taking into account air pollutant concentrations at a reference site located in the regional background, and adjusting for hour of the week, public holidays, season, and wind direction. The LEZ in Berlin, and the LEZ in combination with the heavy-duty vehicle (HDV) transit ban in Munich significantly reduced the PM concentrations, at both traffic sites (TS) and urban background sites (UB). The effects were greater in LEZ stage 3 than in LEZ stages 2 and 1. Moreover, compared with PM, LEZ was more efficient in reducing EC, a component that is considered more toxic than PM mass. In contrast, the LEZ had no consistent effect on NO levels: no effects were observed in Berlin; in Munich, the combination of the LEZ and the HDV transit ban reduced NO at UB site in LEZ stage 1, but without further reductions in subsequent stages of the LEZ. Overall, our study indicated that LEZs, which target the major primary air pollution source in the highly populated city center could be an effective way to improve urban air quality such as PM mass concentration and EC level.