Crowdsourcing Morphology Assessments in Oculoplastic Surgery: Reliability and Validity of Lay People Relative to Professional Image Analysts and Experts.

PURPOSE

To determine if crowdsourced ratings of oculoplastic surgical outcomes provide reliable information compared to professional graders and oculoplastic experts.

METHODS

In this prospective psychometric evaluation, a scale for the rating of postoperative eyelid swelling was constructed using randomly selected images and topic experts. This scale was presented adjacent to 205 test images, including 10% duplicates. Graders were instructed to match the test image to the reference image it most closely resembles. Three sets of graders were solicited: crowdsourced lay people from Amazon Mechanical Turk marketplace, professional graders from the Doheny Image Reading Center (DIRC), and American Society of Ophthalmic Plastic and Reconstructive Surgery surgeons. Performance was assessed by classical correlational analysis and generalizability theory.

RESULTS

The correlation between scores on the first rating and the second rating for the 19 repeated occurrences was 0.60 for lay observers, 0.80 for DIRC graders and 0.84 for oculoplastic experts. In terms of inter-group rating reliability for all photos, the scores provided by lay observers were correlated with DIRC graders at a level of r = 0.88 and to experts at r = 0.79. The pictures themselves accounted for the greatest amount of variation among all groups. The amount of variation in the scores due to the rater was highest in the lay group at 25%, and was 20% and 21% for DIRC graders and experts, respectively.

CONCLUSIONS

Crowdsourced observers are insufficiently precise to replicate the results of experts in grading postoperative eyelid swelling. DIRC graders performed similarly to experts and present a less resource-intensive option.