School of Computing, Dublin City University, Dublin, Ireland.
ADAPT Centre, School of Computer Science, University College Dublin, Dublin, Ireland.
J Biomed Semantics. 2023 Aug 31;14(1):12. doi: 10.1186/s13326-023-00293-9.
This paper proposes Cyrus, a new transparency evaluation framework, for Open Knowledge Extraction (OKE) systems. Cyrus is based on the state-of-the-art transparency models and linked data quality assessment dimensions. It brings together a comprehensive view of transparency dimensions for OKE systems. The Cyrus framework is used to evaluate the transparency of three linked datasets, which are built from the same corpus by three state-of-the-art OKE systems. The evaluation is automatically performed using a combination of three state-of-the-art FAIRness (Findability, Accessibility, Interoperability, Reusability) assessment tools and a linked data quality evaluation framework, called Luzzu. This evaluation includes six Cyrus data transparency dimensions for which existing assessment tools could be identified. OKE systems extract structured knowledge from unstructured or semi-structured text in the form of linked data. These systems are fundamental components of advanced knowledge services. However, due to the lack of a transparency framework for OKE, most OKE systems are not transparent. This means that their processes and outcomes are not understandable and interpretable. A comprehensive framework sheds light on different aspects of transparency, allows comparison between the transparency of different systems by supporting the development of transparency scores, gives insight into the transparency weaknesses of the system, and ways to improve them. Automatic transparency evaluation helps with scalability and facilitates transparency assessment. The transparency problem has been identified as critical by the European Union Trustworthy Artificial Intelligence (AI) guidelines. In this paper, Cyrus provides the first comprehensive view of transparency dimensions for OKE systems by merging the perspectives of the FAccT (Fairness, Accountability, and Transparency), FAIR, and linked data quality research communities.
In Cyrus, data transparency includes ten dimensions which are grouped in two categories. In this paper, six of these dimensions, i.e., provenance, interpretability, understandability, licensing, availability, interlinking have been evaluated automatically for three state-of-the-art OKE systems, using the state-of-the-art metrics and tools. Covid-on-the-Web is identified to have the highest mean transparency.
This is the first research to study the transparency of OKE systems that provides a comprehensive set of transparency dimensions spanning ethics, trustworthy AI, and data quality approaches to transparency. It also demonstrates how to perform automated transparency evaluation that combines existing FAIRness and linked data quality assessment tools for the first time. We show that state-of-the-art OKE systems vary in the transparency of the linked data generated and that these differences can be automatically quantified leading to potential applications in trustworthy AI, compliance, data protection, data governance, and future OKE system design and testing.
本文提出了 Cyrus，这是一个用于开放知识抽取（OKE）系统的新透明度评估框架。Cyrus 基于最先进的透明度模型和关联数据质量评估维度。它汇集了 OKE 系统透明度维度的综合视图。Cyrus 框架用于评估三个关联数据集的透明度，这些数据集由三个最先进的 OKE 系统从同一个语料库构建而成。评估是使用三种最先进的 FAIRness（可发现性、可访问性、互操作性、可重用性）评估工具和一个称为 Luzzu 的关联数据质量评估框架的组合自动执行的。这项评估包括六个 Cyrus 数据透明度维度，对于这些维度，可以确定现有的评估工具。OKE 系统以关联数据的形式从非结构化或半结构化文本中提取结构化知识。这些系统是高级知识服务的基本组成部分。但是，由于缺乏 OKE 的透明度框架，大多数 OKE 系统都不透明。这意味着它们的过程和结果不可理解和可解释。一个全面的框架揭示了透明度的不同方面，通过支持透明度分数的开发，可以比较不同系统的透明度，深入了解系统的透明度弱点以及改进它们的方法。自动透明度评估有助于可扩展性并促进透明度评估。透明度问题已被欧洲联盟可信人工智能 (AI) 准则确定为关键问题。在本文中，Cyrus 通过合并 FAccT（公平、问责制和透明度）、FAIR 和关联数据质量研究社区的观点，为 OKE 系统提供了透明度维度的全面视图。
在 Cyrus 中，数据透明度包括十个维度，分为两类。在本文中，使用最先进的指标和工具，自动评估了这三个最先进的 OKE 系统的六个维度，即来源、可解释性、可理解性、许可、可用性和互链接。Covid-on-the-Web 被确定为具有最高平均透明度。
这是第一项研究 OKE 系统透明度的研究，它提供了一套全面的透明度维度，涵盖了道德、可信 AI 和数据质量方法的透明度。它还演示了如何首次结合现有的 FAIRness 和关联数据质量评估工具进行自动透明度评估。我们表明，最先进的 OKE 系统在生成的关联数据的透明度上存在差异，并且这些差异可以自动量化，从而为可信 AI、合规性、数据保护、数据治理以及未来的 OKE 系统设计和测试中的潜在应用提供了可能性。
