We investigated the frequency of affective symptoms in Korean adults with epilepsy who visited epilepsy clinics at two tertiary care hospitals and in healthy adults. We also examined the psychosocial impact of affective symptoms on people with epilepsy (PWE). Participants were asked to complete self-report questionnaires to assess depression and anxiety symptoms, felt stigma, suicidal ideation, and quality of life (QOL). Of 568 PWE, 30.5% exhibited affective symptoms. The frequencies of depression and anxiety symptoms were 27.8% and 15.3%, respectively, significantly higher than those in healthy controls. Those with poor seizure control were more likely to endorse affective symptoms at the time of study. The frequencies of felt stigma and suicidal ideation were higher in PWE with affective symptoms than in those without. Quality of life was impacted by affective symptoms, especially when depression and anxiety coexisted. Reducing affective symptoms by appropriate seizure control may ameliorate psychosocial problems in PWE.