Zhao Wentao, Zhou Dalin, Qiu Xinguo, Jiang Wei
College of Mechanical Engineering, Zhejiang University of Technology, Hangzhou 310023, China.
School of Intelligent Transportation, Zhejiang Institute of Mechanical & Electrical Engineering, Hangzhou 310053, China.
Sensors (Basel). 2021 Mar 10;21(6):1940. doi: 10.3390/s21061940.
The goal of large-scale automatic paintings analysis is to classify and retrieve images using machine learning techniques. The traditional methods use computer vision techniques on paintings to enable computers to represent the art content. In this work, we propose using a graph convolutional network and artistic comments rather than the painting color to classify type, school, timeframe and author of the paintings by implementing natural language processing (NLP) techniques. First, we build a single artistic comment graph based on co-occurrence relations and document word relations and then train an art graph convolutional network (ArtGCN) on the entire corpus. The nodes, which include the words and documents in the topological graph are initialized using a one-hot representation; then, the embeddings are learned jointly for both words and documents, supervised by the known-class training labels of the paintings. Through extensive experiments on different classification tasks using different input sources, we demonstrate that the proposed methods achieve state-of-art performance. In addition, ArtGCN can learn word and painting embeddings, and we find that they have a major role in describing the labels and retrieval paintings, respectively.
大规模自动绘画分析的目标是使用机器学习技术对图像进行分类和检索。传统方法在绘画上运用计算机视觉技术，使计算机能够呈现艺术内容。在这项工作中，我们提议通过实施自然语言处理（NLP）技术，使用图卷积网络和艺术评论而非绘画颜色来对绘画的类型、流派、时间框架和作者进行分类。首先，我们基于共现关系和文档词关系构建一个单一的艺术评论图，然后在整个语料库上训练一个艺术图卷积网络（ArtGCN）。拓扑图中的节点包括单词和文档，使用独热表示法进行初始化；然后，在绘画的已知类别训练标签的监督下，共同学习单词和文档的嵌入。通过使用不同输入源对不同分类任务进行广泛实验，我们证明所提出的方法取得了领先的性能。此外，ArtGCN可以学习单词和绘画嵌入，并且我们发现它们分别在描述标签和检索绘画方面发挥着重要作用。
