Introduction Poor sleep quality can predict poor health and is associated with mortality risk. Many factors are associated with sleep quality such as gender, health, education, socioeconomic status, and stress. The objective of this study was to estimate the magnitude of poor sleep quality among visitors of Primary Healthcare Centers (PHCCs) in Al-Ahsa and to identify factors associated with poor sleep quality. Methods This is an analytical cross-sectional study. A multistage cluster sampling technique was used to recruit 461 visitors to PHCCs in Al-Ahsa Governorate in the Eastern Province of Saudi Arabia. A structured questionnaire was administered through face-to-face interviews. The questionnaire includes demographics, a validated Arabic version of the Pittsburgh Sleep Quality Index (PSQI), the Arabic version of the International Physical Activity Questionnaire (IPAQ), the Arabic version of the Patient Health Questionnaire-2 (PHQ-2), the Arabic version of the Generalized Anxiety Disorder-2 (GAD-2), the Arabic version of Perceived Stress Scale-10 (PSS-10), and a translated Mobile Related Sleep Risk Factors (MRSRF). Univariate analysis was performed using the Mann-Whitney U test for continuous data, the chi-square test (χ²) or Fishers's exact test (as appropriate) for categorical data, and logistic regression for multivariable analysis. A P-value of less than or equal to 0.05 was considered significant. Results The study included 433 participants, with 72.5% of them being poor sleepers (PSQI global score of over 5). The highest percentage of poor sleepers was found among those aged 18 - 28 years (81.7%), with no significant difference between genders (p = 0.676). The study's multivariable logistic regression analysis revealed that poor sleep is associated with smoking four hours before bedtime (OR = 2.9, CI = 1.2 - 6.7), consuming caffeine (drinks or pills) three hours before sleep (OR = 2.3, CI = 1.23 - 4.12) or immediately before bedtime (OR = 3.2, CI = 1.02 - 9.9), using mobile phones right before bedtime (OR = 2.6, CI = 1.5 - 4.5), having anxiety (OR = 5.8, CI = 1.3 - 26.2), and depression symptoms (OR = 6.5, CI = 2.9 - 14.5), among other risk factors. Conclusion The prevalence of poor sleep quality in our sample was notably high at 72.5%. Many factors are strongly associated with poor sleep quality including experiencing symptoms of anxiety and depression. Longitudinal studies are needed to explore this crucial health issue further. Healthcare providers in Al-Ahsa should pay particular attention while assessing patients who suffer from sleep disturbance by screening them for depression and anxiety and raising public awareness of the importance of good quality sleep and the factors that affect it.