IEEE Trans Pattern Anal Mach Intell. 2022 Mar;44(3):1304-1319. doi: 10.1109/TPAMI.2020.3024207. Epub 2022 Feb 3.
Image cropping aims to improve the composition as well as aesthetic quality of an image by removing extraneous content from it. Most of the existing image cropping databases provide only one or several human-annotated bounding boxes as the groundtruths, which can hardly reflect the non-uniqueness and flexibility of image cropping in practice. The employed evaluation metrics such as intersection-over-union cannot reliably reflect the real performance of a cropping model, either. This work revisits the problem of image cropping, and presents a grid anchor based formulation by considering the special properties and requirements (e.g., local redundancy, content preservation, aspect ratio) of image cropping. Our formulation reduces the searching space of candidate crops from millions to no more than ninety. Consequently, a grid anchor based cropping benchmark is constructed, where all crops of each image are annotated and more reliable evaluation metrics are defined. To meet the practical demands of robust performance and high efficiency, we also design an effective and lightweight cropping model. By simultaneously considering the region of interest and region of discard, and leveraging multi-scale information, our model can robustly output visually pleasing crops for images of different scenes. With less than 2.5M parameters, our model runs at a speed of 200 FPS on one single GTX 1080Ti GPU and 12 FPS on one i7-6800K CPU. The code is available at: https://github.com/HuiZeng/Grid-Anchor-based-Image-Cropping-Pytorch.
图像裁剪旨在通过从图像中去除多余的内容来改善图像的构图和美学质量。大多数现有的图像裁剪数据库仅提供一个或几个人工注释的边界框作为groundtruths，这很难反映实际裁剪中边界框的非唯一性和灵活性。所采用的评价指标（如交并比）也不能可靠地反映裁剪模型的实际性能。本工作重新审视了图像裁剪问题，并提出了一种基于网格锚点的公式，同时考虑了图像裁剪的特殊性质和要求（例如，局部冗余、内容保留、纵横比）。我们的公式将候选裁剪的搜索空间从数百万减少到不超过九十。因此，构建了一个基于网格锚点的裁剪基准，其中每个图像的所有裁剪都进行了注释，并定义了更可靠的评价指标。为了满足鲁棒性能和高效率的实际需求，我们还设计了一种有效且轻量级的裁剪模型。通过同时考虑感兴趣区域和丢弃区域，并利用多尺度信息，我们的模型可以为不同场景的图像稳健地输出视觉上令人愉悦的裁剪。我们的模型参数少于 250 万，在单个 GTX 1080Ti GPU 上的运行速度为 200 FPS，在单个 i7-6800K CPU 上的运行速度为 12 FPS。代码可在：https://github.com/HuiZeng/Grid-Anchor-based-Image-Cropping-Pytorch 找到。
