Recent studies show that rare, deleterious variants (RDVs) in certain genes are critical determinants of heritable cancer risk. To more comprehensively understand RDVs, we performed the largest-to-date germline variant calling analysis in a case-control setting for a multi-cancer association study from whole-exome sequencing data of 20,789 participants, split into discovery and validation cohorts. We confirm and extend known associations between cancer risk and germline RDVs in specific gene-sets, including DNA repair (OR = 1.50; p-value = 8.30e-07; 95% CI: 1.28-1.77), cancer predisposition (OR = 1.51; p-value = 4.58e-08; 95% CI: 1.30-1.75), and somatic cancer drivers (OR = 1.46; p-value = 4.04e-06; 95% CI: 1.24-1.72). Furthermore, personal RDV load in these gene-sets associated with increased risk, younger age of onset, increased M1 macrophages in tumor and, increased tumor mutational burden in specific cancers. Our findings can be used towards identifying high-risk individuals, who can then benefit from increased surveillance, earlier screening, and treatments that exploit their tumor characteristics, improving prognosis.