From the Department of Radiology, Charité-Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Charitéplatz 1, 10117 Berlin, Germany (U.G., P.J.); Data Analytics and Computational Statistics, Hasso Plattner Institute, Digital Engineering Faculty, University of Potsdam, Potsdam, Germany (U.G.); and Berlin Institute of Health, Berlin, Germany (P.J.).
Radiology. 2022 Jun;303(3):524-530. doi: 10.1148/radiol.211832. Epub 2022 Mar 8.
Background Current software applications for human observer studies of images lack flexibility in study design, platform independence, multicenter use, and assessment methods and are not open source, limiting accessibility and expandability. Purpose To develop a user-friendly software platform that enables efficient human observer studies in medical imaging with flexibility of study design. Materials and Methods Software for human observer imaging studies was designed as an open-source web application to facilitate access, platform-independent usability, and multicenter studies. Different interfaces for study creation, participation, and management of results were implemented. The software was evaluated in human observer experiments between May 2019 and March 2021, in which duration of observer responses was tracked. Fourteen radiologists evaluated and graded software usability using the 100-point system usability scale. The application was tested in Chrome, Firefox, Safari, and Edge browsers. Results Software function was designed to allow visual grading analysis (VGA), multiple-alternative forced-choice (-AFC), receiver operating characteristic (ROC), localization ROC, free-response ROC, and customized designs. The mean duration of reader responses per image or per image set was 6.2 seconds ± 4.8 (standard deviation), 5.8 seconds ± 4.7, 8.7 seconds ± 5.7, and 6.0 seconds ± 4.5 in four-AFC with 160 image quartets per reader, four-AFC with 640 image quartets per reader, localization ROC, and experimental studies, respectively. The mean system usability scale score was 83 ± 11 (out of 100). The documented code and a demonstration of the application are available online (). Conclusion A user-friendly and efficient open-source application was developed for human reader experiments that enables study design versatility, as well as platform-independent and multicenter usability. © RSNA, 2022 See also the editorial by Thompson in this issue.
背景 当前用于图像人工观察者研究的软件应用程序在研究设计、平台独立性、多中心使用和评估方法方面缺乏灵活性，并且不是开源的，这限制了它们的可访问性和可扩展性。
目的 开发一个用户友好的软件平台，以实现医学成像中的高效人工观察者研究，并具有灵活的研究设计。
材料和方法 设计了用于医学成像观察者研究的软件，作为一个开源的网络应用程序，以促进访问、平台独立的可用性和多中心研究。实现了用于创建研究、参与和管理结果的不同界面。该软件于 2019 年 5 月至 2021 年 3 月在人类观察者实验中进行了评估，其中跟踪了观察者响应的持续时间。14 名放射科医生使用 100 分制系统可用性量表评估和分级软件可用性。该应用程序在 Chrome、Firefox、Safari 和 Edge 浏览器中进行了测试。
结果 软件功能旨在允许视觉分级分析 (VGA)、多项迫选 (m-AFC)、受试者工作特征 (ROC)、定位 ROC、自由响应 ROC 和定制设计。在四项 AFC 中，每位读者 160 个图像四分位数，四项 AFC 中每位读者 640 个图像四分位数，定位 ROC 和实验研究中，每位读者每个图像或每个图像集的平均读者响应时间分别为 6.2 秒±4.8(标准差)、5.8 秒±4.7、8.7 秒±5.7 和 6.0 秒±4.5。系统可用性量表的平均得分为 83±11(满分 100)。可在线获取记录的代码和应用程序演示()。
结论 开发了一个用户友好且高效的开源应用程序，用于人类读者实验，该程序可实现研究设计的多功能性以及平台独立性和多中心可用性。
