Does recollection error threaten the validity of cross-sectional studies of effectiveness?

New cross-sectional studies have been designed to evaluate therapeutic effectiveness of medical and surgical treatments. The extent to which error in recollection may threaten the validity of conclusions reached in these studies has not been determined. The purpose of this research was to evaluate the impact of recollection error by comparing patients' prospectively acquired reports about their condition before total hip replacement with their recollections of their preoperative condition obtained several years after surgery. A total of 104 patients prospectively completed the Hip Rating Questionnaire (HRQ), a valid, reproducible, responsive, disease-specific scale composed of four domains (pain, walking, function, and impact of hip arthritis on overall health). These same patients then completed the HRQ several years after surgery by recalling their preoperative condition. Current postoperative condition was also obtained several years after surgery with the HRQ. Patient characteristics include: 55% were women, mean age was 67 years, 90% had osteoarthritis, 78% had no prognostically significant comorbid disease, and the mean time interval between surgery and recall was 2.5 years. Comparison of prospective and recalled responses with the weighted kappa and intraclass correlation coefficients showed poor to fair agreement in three domains, and moderate agreement in the fourth domain. Overall, the directions of the recollection errors were toward patients' recalling more pain, better walking, better function, and worse impact of hip arthritis on health than they reported before surgery. When the data were stratified to determine if there were systematic biases among major patient subgroups, there were discrepancies in the percentage of patients within each subgroup who had recollection error for the different domains, as well as differences in the magnitudes and directions of the recollection errors. These results indicate that relying on patients' recollections does not provide an accurate measure of preoperative state, and that attempting to adjust data is not feasible because the directions and magnitudes of recollection error vary for major subgroups of patients. In addition, when outcome was assessed using postoperative HRQ responses, the cross-sectional data overestimated the effectiveness of total hip replacement in 68% of patients. It is concluded that cross-sectional data do not accurately portray baseline preintervention condition and therefore can lead to overestimating, as in this instance, or to underestimating effectiveness.