In contrast to natural antibodies that rely mainly on the heavy chain to establish contacts with their cognate antigen, we have developed a bispecific antibody format in which the light chain (LC) drives antigen binding and specificity. To better understand epitope-paratope interactions in this context, we determined the X-ray crystallographic structures of an antigen binding fragment (Fab) in complex with human CD47 and another Fab in complex with human PD-L1. These Fabs contain a κ-LC and a λ-LC, respectively, which are paired with an identical heavy chain (HC). The structural analysis of these complexes revealed the dominant contribution of the LCs to antigen binding, but also that the common HC provides some contacts in both CD47 and PD-L1 Fab complexes. The anti-CD47 Fab was affinity optimized by diversifying complementary-determining regions of the LC followed by phage display selections. Using homology modeling, the contributions of the amino acid modification to the affinity increase were analyzed. Our results demonstrate that, despite a less prominent role in natural antibodies, the LC can mediate high affinity binding to different antigens and neutralize their biological function. Importantly, Fabs containing a common variable heavy (VH) domain enable the generation of bispecific antibodies retaining a truly native structure, maximizing their therapeutic potential.