The basic steps in building up language involve binding words of different categories into a hierarchical structure. To what extent these steps are universal or differ across languages is an open issue. Here we examine the neural dynamics of phrase structure building in Chinese-a language that in contrast to other languages heavily depends on contextual semantic information. We used functional magnetic resonance imaging and dynamic causal modeling to identify the relevant brain regions and their dynamic relations. Language stimuli consisted of syntax-driving determiners, semantics-embedded classifiers, and nonverbal symbols making up for two-component sequences manipulated by the factors structure (phrase/list) and number of words (2-word/1-word). Processing phrases compared with word lists elicited greater activation in the anterior part of Broca's area, Brodmann area (BA) 45, and the left posterior superior/middle temporal gyri (pSTG/pMTG), while processing two words against one word led to stronger involvement of the left BA 45, BA 44, and insula. Differential network modulations emerging from subparts of Broca's area revealed that phrasal construction in particular highly modulated the direct connection from BA 44 to left pMTG, suggesting BA 44's primary role in phrase structure building. Conversely, the involvement of BA 45 rather appears sensitive to the reliance on lexico-semantic information in Chinese. Against the background of previous findings from other languages, the present results indicate that phrase structure building has a universal neural basis within the left fronto-temporal network. Most importantly, they provide the first evidence demonstrating that the structure-building network may be modulated by language-specific characteristics.