Extant phenotypic and genetic literature has established consistent relationships between autoimmune, autoinflammatory, and psychiatric disorders. However, a comprehensive model investigating the association between a broad range of psychiatric disorders and immune-mediated disease in a multivariate framework is lacking. We utilized Genomic Structural Equation Modeling (Genomic SEM) to establish a factor structure across 11 immune-mediated diseases. Genetic correlations between these immune factors were examined with five established factors across 13 psychiatric disorders. We develop and validate a new heterogeneity metric, Q, that quantifies the degree to which factor correlations are driven by more specific pairwise associations, which were further investigated in the form of residual genetic correlations. A four-factor model of immune-mediated diseases fit the data well and described a continuum from autoimmune to autoinflammatory diseases, reflecting autoimmune, celiac, mixed pattern, and autoinflammatory diseases. Analyses revealed six significant factor correlations between the immune and psychiatric factors, including autoimmune and mixed pattern diseases with the substance use factors, autoinflammatory diseases and mixed pattern diseases with the internalizing factor, autoinflammatory diseases with the compulsive disorders factor, and autoinflammatory diseases with the schizophrenia/bipolar factor. Additionally, we find evidence of divergence in associations within factors as indicated by Q and 10 significant residual genetic correlations between individual psychiatric disorders and immune-mediated diseases. The results suggest that previously described relationships between specific psychiatric disorders and immune-mediated diseases often capture broader pathways of risk sharing indexed by our genomic factors yet are more specific than a general association across all psychiatric disorders and immune-mediated diseases.