Hamilton-Fletcher Giles, Wright Thomas D, Ward Jamie
Multisens Res. 2016;29(4-5):337-63. doi: 10.1163/22134808-00002519.
Visual sensory substitution devices (SSDs) can represent visual characteristics through distinct patterns of sound, allowing a visually impaired user access to visual information. Previous SSDs have avoided colour and when they do encode colour, have assigned sounds to colour in a largely unprincipled way. This study introduces a new tablet-based SSD termed the ‘Creole’ (so called because it combines tactile scanning with image sonification) and a new algorithm for converting colour to sound that is based on established cross-modal correspondences (intuitive mappings between different sensory dimensions). To test the utility of correspondences, we examined the colour–sound associative memory and object recognition abilities of sighted users who had their device either coded in line with or opposite to sound–colour correspondences. Improved colour memory and reduced colour-errors were made by users who had the correspondence-based mappings. Interestingly, the colour–sound mappings that provided the highest improvements during the associative memory task also saw the greatest gains for recognising realistic objects that also featured these colours, indicating a transfer of abilities from memory to recognition. These users were also marginally better at matching sounds to images varying in luminance, even though luminance was coded identically across the different versions of the device. These findings are discussed with relevance for both colour and correspondences for sensory substitution use.
视觉感官替代设备（SSDs）可以通过独特的声音模式来呈现视觉特征，从而让视障用户获取视觉信息。以往的SSD设备都避免使用颜色，即便对颜色进行编码时，也大多采用毫无原则的方式为颜色分配声音。本研究推出了一款名为“克里奥尔语”的新型平板SSD设备（之所以这样命名，是因为它将触觉扫描与图像声音化相结合），以及一种基于已确立的跨模态对应关系（不同感官维度之间的直观映射）将颜色转换为声音的新算法。为了测试对应关系的效用，我们考察了使用该设备的视力正常用户的颜色-声音联想记忆和物体识别能力，这些用户所使用的设备在声音-颜色对应关系上，有的是按照对应关系编码，有的则相反。基于对应关系进行映射的用户在颜色记忆方面有所改善，颜色错误也有所减少。有趣的是，在联想记忆任务中带来最大改善的颜色-声音映射，在识别具有这些颜色的逼真物体时也有最大的提升，这表明能力从记忆转移到了识别。即便在设备的不同版本中亮度编码相同，这些用户在将声音与亮度不同的图像进行匹配时也略胜一筹。我们将结合颜色和感官替代用途的对应关系来讨论这些发现。
