Sexual quality of life is important for many individuals; therefore, this study examined the frequency of depression in a younger and middle-aged population-based sample, and the impact on physical activity, quality of life, and sexual activity, and the role of antidepressant and cardiac drugs on sexual function. The sample, ages 20 to 59 years, completed relevant items for depression, sexual activity, physical activity, quality of life, and drugs. Data were analyzed using descriptive statistics, Pearson correlations, t-tests, and analysis of variance. Those sexually active had lower mean depression scores and higher ratings of quality of life, while those sexually inactive had greater depression and took more drugs. Thorough assessment of drugs, depression, cardiac and other medical conditions, and interest and engagement in sexual activity should be routinely assessed in all patients, regardless of age.