A series of naphthalene diimide (NDI) based n-type conjugated polymers with amino-functionalized side groups and backbones were synthesized and used as cathode interlayers (CILs) in polymer and perovskite solar cells. Because of controllable amine side groups, all the resulting polymers exhibited distinct electronic properties such as oxidation potential of side chains, charge carrier mobilities, self-doping behaviors, and interfacial dipoles. The influences of the chemical variation of amine groups on the cathode interfacial effects were further investigated in both polymer and perovskite solar cells. We found that the decreased electron-donating property and enhanced steric hindrance of amine side groups substantially weaken the capacities of altering the work function of the cathode and trap passivation of the perovskite film, which induced ineffective interfacial modifications and declining device performance. Moreover, with further improvement of the backbone design through the incorporation of a rigid acetylene spacer, the resulting polymers substantially exhibited an enhanced electron-transporting property. Upon use as CILs, high power conversion efficiencies (PCEs) of 10.1% and 15.2% were, respectively, achieved in polymer and perovskite solar cells. Importantly, these newly developed n-type polymers were allowed to be processed over a broad thickness range of CILs in photovoltaic devices, and a prominent PCE of over 8% for polymer solar cells and 13.5% for perovskite solar cells can be achieved with the thick interlayers over 100 nm, which is beneficial for roll-to-roll coating processes. Our findings contribute toward a better understanding of the structure-performance relationship between CIL material design and solar cell performance, and provide important insights and guidelines for the design of high-performance n-type CIL materials for organic and perovskite optoelectronic devices.