University of California, Los Angeles, Department of Bioengineering, 924 Westwood Boulevard, Suite 420, Los Angeles, CA, 90024, USA.
University of California, Los Angeles, Department of Bioengineering, 924 Westwood Boulevard, Suite 420, Los Angeles, CA, 90024, USA.
Comput Biol Med. 2018 Jan 1;92:55-63. doi: 10.1016/j.compbiomed.2017.10.034. Epub 2017 Nov 3.
It is crucial for clinicians to stay up to date on current literature in order to apply recent evidence to clinical decision making. Automatic summarization systems can help clinicians quickly view an aggregated summary of literature on a topic. Casama, a representation and summarization system based on "contextualized semantic maps," captures the findings of biomedical studies as well as the contexts associated with patient population and study design. This paper presents a user-oriented evaluation of Casama in comparison to a context-free representation, SemRep.
The effectiveness of the representation was evaluated by presenting users with manually annotated Casama and SemRep summaries of ten articles on driver mutations in cancer. Automatic annotations were evaluated on a collection of articles on EGFR mutation in lung cancer. Seven users completed a questionnaire rating the summarization quality for various topics and applications.
Casama had higher median scores than SemRep for the majority of the topics (p≤ 0.00032), all of the applications (p≤ 0.00089), and in overall summarization quality (p≤ 1.5e-05). Casama's manual annotations outperformed Casama's automatic annotations (p = 0.00061).
Casama performed particularly well in the representation of strength of evidence, which was highly rated both quantitatively and qualitatively. Users noted that Casama's less granular, more targeted representation improved usability compared to SemRep.
This evaluation demonstrated the benefits of a contextualized representation for summarizing biomedical literature on cancer. Iteration on specific areas of Casama's representation, further development of its algorithms, and a clinically-oriented evaluation are warranted.
临床医生及时了解当前文献对于将最新证据应用于临床决策至关重要。自动摘要系统可以帮助临床医生快速查看关于某个主题的文献汇总摘要。Casama 是一种基于“上下文语义图”的表示和摘要系统，它可以捕获生物医学研究的结果以及与患者人群和研究设计相关的上下文。本文通过与无上下文表示 SemRep 进行对比，对面向用户的 Casama 进行评估。
通过向用户展示十篇关于癌症驱动突变的文章的 Casama 和 SemRep 手动标注摘要，评估表示的有效性。还对一组关于肺癌中 EGFR 突变的文章进行了自动标注评估。七位用户完成了一份问卷，针对各种主题和应用程序对摘要质量进行评分。
在大多数主题（p≤0.00032）、所有应用程序（p≤0.00089）以及整体摘要质量（p≤1.5e-05）方面，Casama 的中位数评分均高于 SemRep。Casama 的手动标注优于 Casama 的自动标注（p=0.00061）。
Casama 在证据强度的表示方面表现尤其出色，无论是在定量还是定性方面都得到了高度评价。用户指出，与 SemRep 相比，Casama 粒度更小、更有针对性的表示提高了可用性。
该评估证明了上下文化表示在总结癌症生物医学文献方面的优势。需要对 Casama 的表示进行特定领域的迭代、进一步开发其算法以及进行临床导向的评估。
