Understanding gene-environment interaction (GxE) is crucial for deciphering the genetic architecture of human complex traits. However, current statistical methods for GxE inference face challenges in both scalability and interpretability. Here we introduce PIGEON-a unified statistical framework for quantifying polygenic GxE using a variance component analytical approach. Based on this framework, we outline the main objectives in GxE studies and introduce an estimation procedure that requires only summary statistics data as input. We demonstrate the effectiveness of PIGEON through theoretical and empirical analyses, including a quasi-experimental gene-by-education study of health outcomes and gene-by-sex interaction for 530 traits using UK Biobank. We also identify genetic interactors that explain the treatment effect heterogeneity in a clinical trial on smoking cessation. PIGEON suggests a path towards polygenic, summary statistics-based inference in future GxE studies.