IEEE Trans Vis Comput Graph. 2022 Jan;28(1):217-226. doi: 10.1109/TVCG.2021.3114848. Epub 2021 Dec 24.
Supporting the translation from natural language (NL) query to visualization (NL2VIS) can simplify the creation of data visualizations because if successful, anyone can generate visualizations by their natural language from the tabular data. The state-of-the-art NL2VIS approaches (e.g., NL4DV and FlowSense) are based on semantic parsers and heuristic algorithms, which are not end-to-end and are not designed for supporting (possibly) complex data transformations. Deep neural network powered neural machine translation models have made great strides in many machine translation tasks, which suggests that they might be viable for NL2VIS as well. In this paper, we present ncNet, a Transformer-based sequence-to-sequence model for supporting NL2VIS, with several novel visualization-aware optimizations, including using attention-forcing to optimize the learning process, and visualization-aware rendering to produce better visualization results. To enhance the capability of machine to comprehend natural language queries, ncNet is also designed to take an optional chart template (e.g., a pie chart or a scatter plot) as an additional input, where the chart template will be served as a constraint to limit what could be visualized. We conducted both quantitative evaluation and user study, showing that ncNet achieves good accuracy in the nvBench benchmark and is easy-to-use.
支持从自然语言 (NL) 查询到可视化 (NL2VIS) 的转换可以简化数据可视化的创建，因为如果成功，任何人都可以通过表格数据用自然语言生成可视化。最先进的 NL2VIS 方法（例如 NL4DV 和 FlowSense）基于语义解析器和启发式算法，它们不是端到端的，也不是为支持（可能）复杂的数据转换而设计的。基于深度神经网络的神经机器翻译模型在许多机器翻译任务中取得了重大进展，这表明它们也可能适用于 NL2VIS。在本文中，我们提出了 ncNet，这是一种基于 Transformer 的序列到序列模型，用于支持 NL2VIS，并进行了一些新颖的可视化感知优化，包括使用注意力强制来优化学习过程，以及可视化感知渲染来生成更好的可视化结果。为了增强机器理解自然语言查询的能力，ncNet 还被设计为可以接受可选的图表模板（例如饼图或散点图）作为附加输入，其中图表模板将作为限制可可视化内容的约束。我们进行了定量评估和用户研究，表明 ncNet 在 nvBench 基准测试中具有良好的准确性，并且易于使用。
