Cossío Fernando, Schurz Haiko, Engström Mathias, Barck-Holst Carl, Tsirikoglou Apostolia, Lundström Claes, Gustafsson Håkan, Smith Kevin, Zackrisson Sophia, Strand Fredrik
Karolinska Institute, Department of Oncology-Pathology, Stockholm, Sweden.
Karolinska University Hospital, Department of Radiology, Stockholm, Sweden.
J Med Imaging (Bellingham). 2023 Nov;10(6):061404. doi: 10.1117/1.JMI.10.6.061404. Epub 2023 Mar 20.
Multiple vendors are currently offering artificial intelligence (AI) computer-aided systems for triage detection, diagnosis, and risk prediction of breast cancer based on screening mammography. There is an imminent need to establish validation platforms that enable fair and transparent testing of these systems against external data.
We developed validation of artificial intelligence for breast imaging (VAI-B), a platform for independent validation of AI algorithms in breast imaging. The platform is a hybrid solution, with one part implemented in the cloud and another in an on-premises environment at Karolinska Institute. Cloud services provide the flexibility of scaling the computing power during inference time, while secure on-premises clinical data storage preserves their privacy. A MongoDB database and a python package were developed to store and manage the data on-premises. VAI-B requires four data components: radiological images, AI inferences, radiologist assessments, and cancer outcomes.
To pilot test VAI-B, we defined a case-control population based on 8080 patients diagnosed with breast cancer and 36,339 healthy women based on the Swedish national quality registry for breast cancer. Images and radiological assessments from more than 100,000 mammography examinations were extracted from hospitals in three regions of Sweden. The images were processed by AI systems from three vendors in a virtual private cloud to produce abnormality scores related to signs of cancer in the images. A total of 105,706 examinations have been processed and stored in the database.
We have created a platform that will allow downstream evaluation of AI systems for breast cancer detection, which enables faster development cycles for participating vendors and safer AI adoption for participating hospitals. The platform was designed to be scalable and ready to be expanded should a new vendor want to evaluate their system or should a new hospital wish to obtain an evaluation of different AI systems on their images.
目前有多家供应商提供基于乳腺钼靶筛查的人工智能（AI）计算机辅助系统，用于乳腺癌的分诊检测、诊断和风险预测。迫切需要建立验证平台，以便针对外部数据对这些系统进行公平、透明的测试。
我们开发了乳腺成像人工智能验证（VAI-B）平台，用于对乳腺成像中的AI算法进行独立验证。该平台是一种混合解决方案，一部分在云端实现，另一部分在卡罗林斯卡学院的本地环境中实现。云服务在推理时提供扩展计算能力的灵活性，而安全的本地临床数据存储则保护了数据隐私。开发了一个MongoDB数据库和一个Python包，用于在本地存储和管理数据。VAI-B需要四个数据组件：放射图像、AI推理、放射科医生评估和癌症结果。
为了对VAI-B进行试点测试，我们根据瑞典国家乳腺癌质量登记处，定义了一个病例对照人群，其中包括8080例被诊断为乳腺癌的患者和36339名健康女性。从瑞典三个地区的医院提取了超过100000次乳腺钼靶检查的图像和放射学评估。这些图像由来自三个供应商的AI系统在虚拟专用云中进行处理，以生成与图像中癌症迹象相关的异常分数。共有105706次检查已被处理并存储在数据库中。
我们创建了一个平台，可对用于乳腺癌检测的AI系统进行下游评估，这为参与的供应商缩短了开发周期，并为参与的医院更安全地采用AI提供了便利。该平台设计为可扩展的，如果新供应商想要评估其系统，或者新医院希望对其图像上的不同AI系统进行评估，该平台随时可以扩展。
