A novel approach to automating context-driven alternative text generation through purposeful games

Abstract

Accessibility of the Web is a pervasive issue, owing to the persistence of accessibility barriers (e.g., poor navigation, lack of/unsuitable alternative text (alt text), complex web forms), with significant impact on users with disabilities. Alt text barriers in particular, are some of the most prevalent web accessibility barriers affecting a wide range of media and are underpinned by a lack of understanding and guidelines on what constitutes suitable alt text. Past work has shown that the ‘context in which the image is used in’ is ironclad for suitability yet loosely defined. Whilst there is a need to automate alt text generation, current solutions disregard context in alt text and are lacklustre with regard to suitability. In this research, an empirical exploratory study that investigates the views of web content creators and visually impaired users on suitability is conducted to bridge the functional gap between experiences and best available practice. The first definition of ‘Alt Text Context’ is proposed providing a systematic way to assess when alt text is necessary and what it should convey. Further, the first crowdsourcing game for context-driven alt text authorship and evaluation—TagALTlong—is presented. TagALTlong’s design is informed by relevant literature, empirical qualitative insights and the proposed definition of context in alt text. Following an empirical user study, 125 non-expert players were recruited to play TagALTlong over a six-week period, resulting in 1208 authored and 1836 rated alt text descriptions, respectively. The resulting dataset was used to fine-tune and train an AI model for automated alt text generation to assess whether average human-level alt text quality can be approximated whilst automating the process. Results indicated the improved performance of the model that was fine-tuned and trained on the GWAP-generated dataset compared to pure image processing, subsequently demonstrating the value of the dataset.