VIIDA and InViDe: computational approaches for generating and evaluating inclusive image paragraphs for the visually impaired.

BACKGROUND

Existing image description methods when used as Assistive Technologies often fall short in meeting the needs of blind or low vision (BLV) individuals. They tend to either compress all visual elements into brief captions, create disjointed sentences for each image region, or provide extensive descriptions.

PURPOSE

To address these limitations, we introduce VIIDA, a procedure aimed at the Visually Impaired which implements an Image Description Approach, focusing on webinar scenes. We also propose InViDe, an Inclusive Visual Description metric, a novel approach for evaluating image descriptions targeting BLV people.

METHODS

We reviewed existing methods and developed VIIDA by integrating a multimodal Visual Question Answering model with Natural Language Processing (NLP) filters. A scene graph-based algorithm was then applied to structure final paragraphs. By employing NLP tools, InViDe conducts a multicriteria analysis based on accessibility standards and guidelines.

RESULTS

Experiments statistically demonstrate that VIIDA generates descriptions closely aligned with image content as well as human-written linguistic features, and that suit BLV needs. InViDe offers valuable insights into the behaviour of the compared methods - among them, state-of-the-art methods based on Large Language Models - across diverse criteria.

CONCLUSION

VIIDA and InViDe emerge as efficient Assistive Technologies, combining Artificial Intelligence models and computational/mathematical techniques to generate and evaluate image descriptions for the visually impaired with low computational costs. This work is anticipated to inspire further research and application development in the domain of Assistive Technologies. Our codes are publicly available at: https://github.com/daniellf/VIIDA-and-InViDe.