Objective Irritable bowel syndrome (IBS) is one of the most common gastrointestinal disorders and is characterized by recurrent abdominal pain or abnormal defecation. This investigation evaluated the relationship between IBS and self-reported quality of life in a community-dwelling population in Japan. Methods For this cross-sectional survey, we enrolled 1002 volunteers who participated in the Iwaki Health Promotion Project in 2013. IBS symptoms were evaluated using the criteria from the Japanese version of the Rome III Questionnaire. The assessments included an interview to obtain sociodemographic data, the second version of the Short-Form Health Survey (SF-36), and the Center for Epidemiologic Studies Depression Scale. Multiple regression analysis was used to assess the relationship between IBS symptoms and scores on the SF-36. Results A total of 59 subjects (5.9%) were classified as having IBS. Scores for all eight domains of the SF-36, the physical component summary, and the mental component summary were significantly and negatively associated with the Center for Epidemiologic Studies Depression scores. Physical functioning, role physical, vitality, mental health, and physical component summary scores were significantly and negatively associated with IBS. Conclusions The burden of IBS symptoms affects both physical and mental wellbeing, even after adjusting for confounders. Our findings suggest that screening for IBS symptoms and evaluating the need for medical care is important for community health workers.