Non-homologous end joining (NHEJ) is the main repair pathway for double-strand DNA breaks (DSBs) in mammals. DNA polymerases lambda (Pol λ) and mu (Pol μ), members of the Pol X family, play a key role in this process. However, their interaction within the NHEJ complexes is unclear. Here, we present cryo-EM structures of Pol λ in complex with the DNA-PK long-range synaptic complex, and Pol μ bound to Ku70/80-DNA. These structures identify interaction sites between Ku70/80 and Pol X BRCT domains. Using mutants at the proteins interface in functional assays including cell transfection with an original gap-filling reporter, we define the role of the BRCT domain in the recruitment and activity of the two Pol X members in NHEJ and in their contribution to cell survival following DSBs. Finally, we propose a unified model for the interaction of all Pol X members with Ku70/80.