Epistasis, or gene-gene interaction, contributes substantially to trait variation in organisms ranging from yeast to humans, and modeling epistasis directly is critical to understanding the genotype-phenotype map. However, inference of genetic interactions is challenging compared to inference of individual allele effects due to low statistical power. Furthermore, genetic interactions can appear inconsistent across different quantitative traits, presenting a challenge for the interpretation of detected interactions. Here we present a method called the Combined Analysis of Pleiotropy and Epistasis (CAPE) that combines information across multiple quantitative traits to infer directed epistatic interactions. By combining information across multiple traits, CAPE not only increases power to detect genetic interactions but also interprets these interactions across traits to identify a single interaction that is consistent across all observed data. This method generates informative, interpretable interaction networks that explain how variants interact with each other to influence groups of related traits. This method could potentially be used to link genetic variants to gene expression, physiological endophenotypes, and higher-level disease traits.