The amygdala is often found to be abnormally recruited in social anxiety disorder (SAD) patients. The question whether amygdala activation is primarily abnormal and affects other brain systems or whether it responds "normally" to an abnormal pattern of information conveyed by other brain structures remained unanswered. To address this question, we investigated a network of effective connectivity associated with the amygdala using Granger causality analysis on resting-state functional MRI data of 22 SAD patients and 21 healthy controls (HC). Implications of abnormal effective connectivity and clinical severity were investigated using the Liebowitz Social Anxiety Scale (LSAS). Decreased influence from inferior temporal gyrus (ITG) to amygdala was found in SAD, while bidirectional influences between amygdala and visual cortices were increased compared to HCs. Clinical relevance of decreased effective connectivity from ITG to amygdala was suggested by a negative correlation of LSAS avoidance scores and the value of Granger causality. Our study is the first to reveal a network of abnormal effective connectivity of core structures in SAD. This is in support of a disregulation in predescribed modules involved in affect control. The amygdala is placed in a central position of dysfunction characterized both by decreased regulatory influence of orbitofrontal cortex and increased crosstalk with visual cortex. The model which is proposed based on our results lends neurobiological support towards cognitive models considering disinhibition and an attentional bias towards negative stimuli as a core feature of the disorder.