Ahmad Jamil, Muhammad Khan, Baik Sung Wook
Department of Software, College of Software and Convergence Technology, Sejong University, Seoul, Republic of Korea.
PLoS One. 2017 Aug 31;12(8):e0183838. doi: 10.1371/journal.pone.0183838. eCollection 2017.
In recent years, image databases are growing at exponential rates, making their management, indexing, and retrieval, very challenging. Typical image retrieval systems rely on sample images as queries. However, in the absence of sample query images, hand-drawn sketches are also used. The recent adoption of touch screen input devices makes it very convenient to quickly draw shaded sketches of objects to be used for querying image databases. This paper presents a mechanism to provide access to visual information based on users' hand-drawn partially colored sketches using touch screen devices. A key challenge for sketch-based image retrieval systems is to cope with the inherent ambiguity in sketches due to the lack of colors, textures, shading, and drawing imperfections. To cope with these issues, we propose to fine-tune a deep convolutional neural network (CNN) using augmented dataset to extract features from partially colored hand-drawn sketches for query specification in a sketch-based image retrieval framework. The large augmented dataset contains natural images, edge maps, hand-drawn sketches, de-colorized, and de-texturized images which allow CNN to effectively model visual contents presented to it in a variety of forms. The deep features extracted from CNN allow retrieval of images using both sketches and full color images as queries. We also evaluated the role of partial coloring or shading in sketches to improve the retrieval performance. The proposed method is tested on two large datasets for sketch recognition and sketch-based image retrieval and achieved better classification and retrieval performance than many existing methods.
近年来，图像数据库正以指数级速度增长，这使得它们的管理、索引和检索极具挑战性。典型的图像检索系统依赖样本图像作为查询。然而，在没有样本查询图像的情况下，也会使用手绘草图。最近触摸屏输入设备的采用使得快速绘制物体的阴影草图以用于查询图像数据库变得非常方便。本文提出了一种基于用户使用触摸屏设备绘制的部分着色草图来提供对视觉信息访问的机制。基于草图的图像检索系统面临的一个关键挑战是应对草图中由于缺乏颜色、纹理、阴影和绘图缺陷而固有的模糊性。为了应对这些问题，我们建议使用增强数据集对深度卷积神经网络（CNN）进行微调，以便在基于草图的图像检索框架中从部分着色的手绘草图中提取用于查询规范的特征。大型增强数据集包含自然图像、边缘图、手绘草图、去色和去纹理图像，这使得CNN能够有效地对以各种形式呈现给它的视觉内容进行建模。从CNN提取的深度特征允许使用草图和全彩色图像作为查询来检索图像。我们还评估了草图中部分着色或阴影对提高检索性能的作用。所提出的方法在两个用于草图识别和基于草图的图像检索的大型数据集上进行了测试，并且比许多现有方法取得了更好的分类和检索性能。
