Wang Yun, Hou Zhitao, Shen Leixian, Wu Tongshuang, Wang Jiaqi, Huang He, Zhang Haidong, Zhang Dongmei
IEEE Trans Vis Comput Graph. 2023 Jan;29(1):1222-1232. doi: 10.1109/TVCG.2022.3209357. Epub 2022 Dec 16.
A key challenge to visualization authoring is the process of getting familiar with the complex user interfaces of authoring tools. Natural Language Interface (NLI) presents promising benefits due to its learnability and usability. However, supporting NLIs for authoring tools requires expertise in natural language processing, while existing NLIs are mostly designed for visual analytic workflow. In this paper, we propose an authoring-oriented NLI pipeline by introducing a structured representation of users' visualization editing intents, called editing actions, based on a formative study and an extensive survey on visualization construction tools. The editing actions are executable, and thus decouple natural language interpretation and visualization applications as an intermediate layer. We implement a deep learning-based NL interpreter to translate NL utterances into editing actions. The interpreter is reusable and extensible across authoring tools. The authoring tools only need to map the editing actions into tool-specific operations. To illustrate the usages of the NL interpreter, we implement an Excel chart editor and a proof-of-concept authoring tool, VisTalk. We conduct a user study with VisTalk to understand the usage patterns of NL-based authoring systems. Finally, we discuss observations on how users author charts with natural language, as well as implications for future research.
可视化创作面临的一个关键挑战是熟悉创作工具复杂的用户界面的过程。自然语言界面（NLI）因其可学习性和可用性而具有显著优势。然而，为创作工具支持自然语言界面需要自然语言处理方面的专业知识，而现有的自然语言界面大多是为可视化分析工作流程设计的。在本文中，我们基于对可视化构建工具的形成性研究和广泛调查，引入一种称为编辑操作的用户可视化编辑意图的结构化表示，提出了一种面向创作的自然语言界面管道。编辑操作是可执行的，因此作为中间层将自然语言解释与可视化应用解耦。我们实现了一个基于深度学习的自然语言解释器，将自然语言话语转换为编辑操作。该解释器可在创作工具之间复用和扩展。创作工具只需将编辑操作映射为特定于工具的操作。为了说明自然语言解释器的用法，我们实现了一个Excel图表编辑器和一个概念验证创作工具VisTalk。我们使用VisTalk进行了一项用户研究，以了解基于自然语言的创作系统的使用模式。最后，我们讨论了关于用户如何用自然语言创作图表的观察结果以及对未来研究的启示。
