IEEE Trans Vis Comput Graph. 2020 Jun;26(6):2168-2179. doi: 10.1109/TVCG.2020.2970512. Epub 2020 Jan 31.
Interaction plays a vital role during visual network exploration as users need to engage with both elements in the view (e.g., nodes, links) and interface controls (e.g., sliders, dropdown menus). Particularly as the size and complexity of a network grow, interactive displays supporting multimodal input (e.g., touch, speech, pen, gaze) exhibit the potential to facilitate fluid interaction during visual network exploration and analysis. While multimodal interaction with network visualization seems like a promising idea, many open questions remain. For instance, do users actually prefer multimodal input over unimodal input, and if so, why? Does it enable them to interact more naturally, or does having multiple modes of input confuse users? To answer such questions, we conducted a qualitative user study in the context of a network visualization tool, comparing speech- and touch-based unimodal interfaces to a multimodal interface combining the two. Our results confirm that participants strongly prefer multimodal input over unimodal input attributing their preference to: 1) the freedom of expression, 2) the complementary nature of speech and touch, and 3) integrated interactions afforded by the combination of the two modalities. We also describe the interaction patterns participants employed to perform common network visualization operations and highlight themes for future multimodal network visualization systems to consider.
交互在视觉网络探索中起着至关重要的作用，因为用户需要同时与视图中的元素（例如节点、链接）和界面控件（例如滑块、下拉菜单）进行交互。特别是随着网络的规模和复杂性的增长，支持多模态输入（例如触摸、语音、笔、注视）的交互式显示具有促进视觉网络探索和分析期间流畅交互的潜力。虽然与网络可视化进行多模态交互似乎是一个很有前途的想法，但仍有许多悬而未决的问题。例如，用户实际上是否更喜欢多模态输入而不是单模态输入，如果是，为什么？它是否使他们能够更自然地交互，还是多种输入模式会使用户感到困惑？为了回答这些问题，我们在网络可视化工具的背景下进行了定性的用户研究，将基于语音和触摸的单模态界面与结合两种方式的多模态界面进行了比较。我们的结果证实，参与者强烈倾向于多模态输入而不是单模态输入，他们将这种偏好归因于：1）表达的自由，2）语音和触摸的互补性，以及 3）两种模式结合带来的集成交互。我们还描述了参与者用于执行常见网络可视化操作的交互模式，并强调了未来多模态网络可视化系统需要考虑的主题。
