Investigating Mixed Visual Joint Attention to Inform Technology Design

Abstract

Joint attention (JA) is a crucial component of social interaction. It relies heavily on visual cues, such as eye gaze and pointing. This creates barriers for blind and visually impaired (BVI) people to engage in JA with sighted peers. Yet, little research has characterised these barriers or the strategies BVI people employ to overcome them. Furthermore, there appears to be little understanding of how sighted people engage in JA with BVI people. Ten BVI adults were interviewed to explore their childhood and adult experiences of JA with sighted people. The interviews revealed a lack of identifiable, meaningful JA experiences. These absences are perceived as voids that block engagement. The voids were identified by participants as being caused by a lack of accessible feedback across the whole JA experience, but most crucially in the initiation of JA. Participants identified proximity and context-dependent speech as elements that minimised the effect of voids. Group settings exacerbate this problem, with participants feeling a keen sense of responsibility to fill those voids. The identification of a lack of accessible signalling between BVI and sighted people to initiate JA drove the focal direction for the remainder of the research project. Initiating JA is the first step for successful social interactions. For sighted people, these initial steps rely predominantly on nonverbal visual cues, such as observing facial expressions or hand gestures. Six BVI and six sighted adults were interviewed regarding their experiences of initiating JA with other BVI and sighted adults. BVI participants again highlighted proximity, familiarity, and context-dependent language as key elements that facilitate the initiation of JA. Difficulty with turn-taking and maintaining situational awareness in groups was highlighted by BVI participants. When with other BVI people, matching communication schemas (learned behavioural patterns) made initiating JA easier. Sighted participants also reported a void of JA experiences with BVI people, citing groups as being particularly problematic, too. They did not, however, realise the extent of their role in creating voids, or how impactful the void was for their BVI peers. These findings determined that deviating from established communication schemas cannot be maintained alongside the cognitive overload experienced when social interactions involve maintaining situational awareness in increasingly complex social environments in mixed visual groups. Short videos of four BVI children with eight sighted parents or carers in JA-rich environments were analysed to understand what mixed-visual dyads did to initiate and maintain JA in optimal environments. Findings established that language is crucial for the independence required to coordinate JA. The BVI children oriented themselves primarily toward the JA object, whereas the sighted adults oriented themselves primarily toward the BVI child. Despite the supportive elements of proximity and familiarity, sighted participants used inaccessible speech and nonverbal cues. BVI children continued to turn away from their partner, keep their heads lowered, and not respond to non-verbal gestures. These findings determined that even in optimal JA conditions, voids occurred because the communication schemas are so different. A case study of one congenitally blind child interacting with sighted peers with and without an AI-powered headset designed to support JA initiation was conducted to determine what supportive and what confounding factors contributed to the initiation of JA in mixed-visual groups. All children were observed to experience cognitive overload as the environment became more dynamic. The blind child was unable to independently maintain situational awareness to such an extent that she attempted to initiate JA to an empty room, unaware that her peers had left. The sighted children were observed to update the blind child in the less complex situations, but this was not maintained as the environment became more dynamic. The findings showed that for JA interdependence to be maintained in mixed visual groups, technology needs to account for the entire JA triad, rather than just the blind child. Drawing together knowledge and insights from empirical studies and theoretical research, a framework is presented that provides a platform for understanding the mechanisms of JA initiation in mixed visual groups. This framework is then used to inform implications for technology design that focus on maintaining both the independence and interdependence required, not only for a reduction of the void but also to facilitate frequent JA opportunities in complex and dynamic environments for all members of a group.

Date of Award	9 Dec 2025
Original language	English
Awarding Institution	University of Bristol
Sponsors	Microsoft Res, Microsoft
Supervisor	Ute B Leonards (Supervisor), Oussama Metatla (Supervisor) & Cecily Morrison (Supervisor)