OBJECTIVES: A web-based survey has been administered annually since 2005 throughout Korea to assess the prevalence of adolescent health risk behaviors among middle and high school students. The aim of this study was to evaluate the test-retest reliability of the Korea Youth Risk Behavior Web-based Survey (KYRBWS) questionnaire. METHODS: A convenience sample of 2298 middle and high school students participated in self-administered questionnaire surveys twice, approximately two weeks apart, in 2008. The percent agreement, kappa statistics, and prevalence rates at the first and second surveys were computed for the core subset of 39 self-reported health risk behavior indices of the KYRBWS. RESULTS: Among 39 indices, seven indices had kappas ≥ 0.81 and all of the indices had kappas ≥ 0.41. Based on non-overlapping 95% confidence intervals, three indices had significantly different prevalence rates between the first and second surveys. In the subgroup analyses by school grade and gender, two indices had significantly different reliability estimates between middle and high school students. There were no significantly different reliability estimates between male and female students, except for one index. CONCLUSIONS: This study demonstrated that the reliability estimates for the KYRBWS questionnaire are varied, but generally reliable over time. The indices with low reliability estimates need to be evaluated further in order to determine whether the indices should be modified or deleted from future versions of the KYRBWS.