Wang Hao, Lin Guosheng, Hoi Steven C H, Miao Chunyan
IEEE Trans Pattern Anal Mach Intell. 2023 Mar;45(3):3363-3377. doi: 10.1109/TPAMI.2022.3181294. Epub 2023 Feb 3.
Food is significant to human daily life. In this paper, we are interested in learning structural representations for lengthy recipes, that can benefit the recipe generation and food cross-modal retrieval tasks. Different from the common vision-language data, here the food images contain mixed ingredients and target recipes are lengthy paragraphs, where we do not have annotations on structure information. To address the above limitations, we propose a novel method to unsupervisedly learn the sentence-level tree structures for the cooking recipes. Our approach brings together several novel ideas in a systematic framework: (1) exploiting an unsupervised learning approach to obtain the sentence-level tree structure labels before training; (2) generating trees of target recipes from images with the supervision of tree structure labels learned from (1); and (3) integrating the learned tree structures into the recipe generation and food cross-modal retrieval procedure. Our proposed model can produce good-quality sentence-level tree structures and coherent recipes. We achieve the state-of-the-art recipe generation and food cross-modal retrieval performance on the benchmark Recipe1M dataset.
食物对人类日常生活至关重要。在本文中，我们感兴趣的是学习冗长食谱的结构表示，这可以有益于食谱生成和食物跨模态检索任务。与常见的视觉语言数据不同，这里的食物图像包含混合食材，且目标食谱是冗长的段落，我们没有关于结构信息的注释。为解决上述限制，我们提出一种新颖的方法来无监督地学习烹饪食谱的句子级树结构。我们的方法在一个系统框架中融合了几个新颖的想法：（1）利用无监督学习方法在训练前获得句子级树结构标签；（2）在从（1）中学习到的树结构标签的监督下，从图像生成目标食谱的树；（3）将学习到的树结构整合到食谱生成和食物跨模态检索过程中。我们提出的模型可以生成高质量的句子级树结构和连贯的食谱。我们在基准Recipe1M数据集上实现了当前最优的食谱生成和食物跨模态检索性能。
