“Easier or Harder, Depending on Who the Hearing Person Is”: Codesigning Videoconferencing Tools for Small Groups with Mixed Hearing Status

3.1.1 Session 1: Questioning Current Practices.

The first session explored how each group uses captions and ideas for technology that could support accessible group communication practices. After introductions, we played the game Twenty Questions³ using automatic captioning⁴ to get groups immediately using and thinking about captions in an engaged conversation and to observe their communication practices to inform questions later in the session. The game's frequent turn-taking and niche vocabulary increased the chances for participants to address captioning breakdowns. Participants played for an average of 17:24 minutes (range: 13:08-25:14), and each group played at least two rounds. Researchers took notes on notable interactions and generated questions about how the group approached conversation during the games.

After Twenty Questions, researchers led a group interview, with reflections on the game and tailored questions about specific aspects of in-game communication. This was followed by questions on the group's background communicating together and practices they use to communicate effectively. Next, we introduced participants to the broader focus of the study: improving the design of captioning tools to support groups in developing accessible communication practices. Researchers inquired about the impact of hearing people's behavior on communication access, and their reactions to the idea of using technology to give feedback about accessible communication practices. Finally, we used the five features proposed by McDonnell et al. [43]—speech rate, volume, caption lag monitoring, speaker identification, speaker overlap warning—to discuss potential captioning tool features and as a basis for collective brainstorming.

3.1.2 Session 2: Feature Sketches.

The second session developed participants’ proposed feature ideas via individual sketching and group discussion. Sessions began with groups reviewing, discussing, and expanding ideas they generated from Session 1. Next, participants spent five to seven minutes sketching out the ideas they felt were most compelling. When finished, participants sent pictures of their sketch(es) via text or email to the research team. We screen-shared these pictures and each participant presented the context and motivation for their ideas. Participants had the opportunity to respond and ask questions about others’ ideas. After all group members had presented, the group reflected on the ideas and identified their top three shared design priorities.

After the session, the research team created video prototypes [63] of each group's top three ideas (12 features total) to provide a more tangible representation of each idea for critique and discussion (see Supplementary Materials for video prototypes). We elected to make video prototypes rather than functional prototypes as they allow participants to assess high-fidelity, dynamic implementations of their ideas before they are built, enabling low-cost iteration or abandonment of designs. The prototypes integrated participants’ specific designs whenever possible. For each group, researchers made three unique videos that showed a feature's design elements and its usage in a simulated conversation. A final, fourth video showed all the features in use during a short round of Twenty Questions played by the research team.

3.1.3 Session 3: Design Review.

Finally, in the last session, each group reviewed both their own and other groups’ design ideas. To begin, we explained that video prototypes “take new design ideas and animate them to demonstrate how these new ideas might look and function.” We stressed that video prototyping “gives us a chance to view how designs work before they've been built, making them easy to iterate on” and encouraged participants to share their “honest opinions.” Each group watched their three video prototypes and shared reactions. We then played the final video showing all features in use during a conversation and invited final reflections. Afterwards, we reviewed other groups’ video prototypes and feature sets, with discussion after each set. Finally, we concluded by asking participants to reflect on the study as a whole and to share any additional thoughts.

3.2.1 Group A.

Group A are cousins who live across the country from each other and communicate frequently using the video messaging app Marco Polo. They communicate orally when meeting synchronously (both in person and online) and use automatic captions for occasional online meetings with their entire family. All four group members identify as white and female, and their average age is 27.8 (range 25-30).

3.2.2 Group B.

Group B meets weekly as colleagues; their work focuses on technology to support DHH people. They communicate orally with Zoom's automatic captions. While our recruitment materials sought groups with up to five members, we opted to include this group of six since they regularly meet. Five group members identify as female, one identifies as male, all are white, and the average age of the group is 53.7 (range 26-67).

3.2.3 Group C.

Group C are friends who know each other through the Deaf community. Camille and Cad are Deaf; Colin is a child of Deaf adults (CODA) and knows American Sign Language (ASL). However, all three have experience using captions while video conferencing. Colin could not attend Study Session 2 at the last minute, so some group discussion of design ideas occurred asynchronously via email. One group member identifies as female, two identify as male, all are white, and the average age of the group is 53.67 (range 42-61).

3.2.4 Group D.

Group D formed around three friends (Daisy, Deanna, David) who know each other through the Deaf community; Dot joined as Daisy's mom, but she does not regularly chat with the others. Daisy, Deanna, and David primarily communicate via ASL; Dot knows some signed English but is not fluent in ASL. Dot at times identified herself as hearing and as having acquired hearing loss. In contrast to our recruitment criteria, not all members use captions when communicating with each other, and Dot had not previously experienced captioned video calls.

While Group D did not meet our recruitment guidelines, we accepted Mack et al.’s [40] invitation to adapt our study design to work with participants with valuable experience who may not fit all study criteria. Group D provides a useful perspective on how captioning might be used by people who prefer sign language and how to facilitate group communication across language barriers. However, we note areas where they may have different needs and requirements for technology than groups who frequently opt to use captioning. Due to technical issues in Study Session 2, design feature sketching and discussion was conducted asynchronously via email. Three group members identify as female, one identifies as male. Three members are white and one is AfroLatina/x and South Asian. Their average age is 48.5 (range 36-65).

4.1.1 Individual Captioning Practices.

Individual participants’ approaches to caption use online varied based on their hearing ability and communication practices. DHH participants balanced using captions alongside their residual hearing and speechreading skills. Some only used captions “if the audio got spotty or something” (Amelia) and used them “part of the time” (Brian), depending on the group's familiarity and if speakers’ camera feeds were available. Barbara described needing to “stare at the captions” to follow a conversation, but that she compensated for captioning errors by using context and her residual hearing, such that her “brain is constantly correcting and not paying attention to those corrections” (Barbara). However, for Bea, motion from automatic captions “would grab my attention and distract me,” making them something she only uses as a backup. For others, captions operate as “the primary feed of information” (Camille). Cad explained that relying entirely on visual information sources (e.g., captioning, facial expressions, signing, chat), makes video calls “twice as hard for us as it is for hearing people.” Notably, participants approached captioning differently depending on their ability to participate in the conversation without it.

For participants in Group D who primarily communicate via sign language, captions were often a secondary source of information during interpreted conversations. Daisy leverages captions as a backup to ensure that, if interpreters encounter difficulties, she “do[esn't] miss too much information” and so that she can “make sure that the interpreter is voicing what I say correctly.” David finds that captions alone are not sufficient for expressive communication, as he “would use sign, I would not type, I would not chat.” However, he runs a third-party automatic captioning app during all video calls so that he does not miss key information if he looks away from a conversation. Despite these limitations, Deanna emphasized that captions do provide value, as when she's “one Deaf person in a group of hearing people …. captioning gives us a way to be involved.”

Hearing participants largely did not report using captions during video calls. Some participants noted that while they may check captions out of curiosity, they usually would not have them running on their screen (Anna, Audrey, Allison). Though Brenda serves as the moderator who works to ensure access during her group's conversations, she noted that she “did not use the captions much–I mean certainly not for communication access.” Colin demonstrated a mode of being more attentive to captions: while he stated he was only “paying attention a little bit” he also described actively waiting to participate until captions had caught up and monitoring them for errors. Even in groups with active attention to accessible norms, it was not assumed that hearing people would be paying attention to or even viewing captions.

4.1.2 Group Communication Practices.

Groups developed specific practices and norms to fit their conversational context and interpersonal relationships. We highlight examples of how each group's context shaped their communication, then identify key takeaways.

Group A highlights a form that access can take within families. Group members have been close since childhood but had not established explicit access norms. They explained, “We never really talked about accessibility for a hard of hearing person because when you grow up with it, it's just – you already have your system down” (Allison). Amelia affirmed that, while access needs to be actively considered with many others in her life, “I'm very comfortable with these ladies and I'm able to understand them very well.” Further, they usually communicated via the app Marco Polo, where users send recorded videos back and forth, which Audrey described as “super nice because you can't speak over each other.” Despite not having in-app captions, Amelia could access Marco Polo via the Live Transcribe functionality on her phone. Group A did not intentionally create accessible norms, but their established relationships left Amelia feeling well-supported.

Group B is comprised of colleagues who work in DHH spaces, and, in contrast to Group A, they actively enact accessibility practices. They have developed explicit conversation rules, including clear turn-taking and monitoring for caption errors, which they share and teach when outsiders join their group. They developed their rules and habits through time and close collaboration. For instance, Brenda reflected on her communication with Barbara:

“I too watch faces … I've sort of learned when [Barbara's] looking at the captions and it looks like she's not understanding something, then I'm immediately reading the captions myself to see ‘oh did it not get it right?’”

This team climate, where hearing and DHH members alike attend to access, was special for Blake: “I was pretty emotional after the first meeting because it was just so inclusive. … It was a really dramatic difference having those set norms … that gives you time to catch up and it gives the captions time too.” Group B created an environment where all members can effectively collaborate through an active commitment to accessibility norms.

Group C reflected on how the expectations of conversation partners impacted access. For instance, when Cad spends time with his wife's non-signing family via Google Meet, “they also have their issues with the captioning and we laugh about it.” Camille pointed out that this becomes possible in “an environment that's more accepting of flaws.” Participants also speculated about how the study game of Twenty Questions would have differed with “10 hearing people who knew nothing about Deaf people in the room … they would be all talking over each other” (Colin). Camille emphasized that, if playing a game with those group dynamics, “I'm sure I would just fade away and not even be a part of it.” However, in Group C, the hearing member, Colin, is a CODA, so he knows how to differentiate “hearing norms, Deaf norms, hearing values in a meeting, Deaf values in a meeting” (Camille). In Group C they “didn't have to say the rules” (Colin), and implicitly understood that Colin would adjust to Deaf norms, rather than Camille and Cad adjusting to hearing standards.

Group D focused on how their access provisions were shaped by their communication partners. For Daisy, captioning is a “less commensurate method” of communication access, but when without an interpreter, she communicated using “a combination of gesturing, and signing, and captioning, and typing.” Deanna explained that her goal in communication was “to be very accommodating and flexible to deal with whoever is there—the point is accessibility.” Yet, group members noted that, in their experience, “hearing people … don't always have a lot of empathy or understanding” (Dot). This led to frustrating interactions, such as a conversation between Deanna and a new family member who refused to write back and forth and insisted on inaccessibly voicing. Group D stressed that accessible communication, especially for signers who aren't fully supported by captioning, requires mutual flexibility.

While all four groups found that access becomes possible when conversation partners work to meet each other's needs, the form of those approaches varied significantly. Participants also described conversations with others outside their group that took different approaches, to varied degrees of success. Ultimately, this demonstrates that there is no one-size-fits-all approach to accessible conversation between DHH and hearing people. Rather tailored, contextual access evolves and is informed by conversation partners’ relationships with each other.

4.1.3 Developing Group Access Norms.

Groups reflected on the positive impacts and the challenges of developing access norms in mixed groups. Participants, DHH and hearing alike, had experiences where collective effort resulted in more comprehensible communication. Barbara found that most of the hearing people she communicates with “understand how to make themselves understood to people and understand what captions are about.” Still, when outsiders join groups that have set rules there is a learning curve, as “it was hard for [them] to slow down” and led to “a stop/start environment” (Bridget). Brenda explained that she actively moderated those conversations, hoping that “when a hearing person also makes those requests and reminders, that it just helps reinforce the need for making sure things are accessible.” Camille reflected on similar dynamics, concluding that conversations could be “easier or harder, depending on who the hearing person is... Are they aware? Are they unaware?” For her, successful, accessible communication requires “sometimes not following the hearing pattern of turn-taking or communication” (Camille). Desired behavior changes can also depend on the context. For example, if Amelia was the only DHH person in a conversation, she was less likely to ask others to adjust, because “if I understand it, then it's fine.” On the other hand, in conversations that include other DHH people, she would “try to make people aware” of communication rules to “support each other” (Amelia).

However, participants noted that behavior change was not always a smooth process. Barbara reflected that it “takes people time and experience to adopt those norms” and that regular practice and reinforcement were critical. Attending to access often requires significant effort from hearing interlocutors. Colin described his experience monitoring captions for errors as he and others spoke: “if it wasn't right, I wanted to fix it, but I also didn't want to jump in and fix it, I wanted to let you try to repair it for yourself if you wanted to. … There's a lot of cognitive load there, and I'm not even the Deaf person.” Likewise, Allison explained that she “can't do two things at once” and monitoring multiple sources of information while speaking was not feasible for her—despite her commitment to communicating accessibly. Therefore, when considering Cad's observation that “there are hearing people who are experienced, who are cognizant and mindful, and there are those who are not,” we must also recognize the complexity involved in learning and applying this expertise.

4.2.1 Codesign Artifacts.

Table 5 shows key frames from each of the video prototypes that researchers developed based on participants’ ideas. These prototypes reflected participants’ discussions across their proposed ideas, searching for convergence in ideas and priorities. For example, Cad drove discussion with his group, saying “I think we are all looking to make it easier to ID the speaker quickly and easily.” Groups assessed their video prototypes in depth and also reviewed other groups’ designs.

4.2.2 Speaker Identity and Overlap.

Quickly identifying speakers and automatically flagging when they overlap were clear priorities for participants, with Groups A, B, and C designing features (Table 5, A3, B2, C3) to address these information gaps. While speaker identity and overlap have both been explored in prior literature (e.g., [18,23,24,35,37,39,47,58]), participants’ experiences demonstrate that ambiguity around crosstalk and speaker identity has yet to be resolved, with key design nuances and social impacts for each feature in group videoconferences.

Though some human-generated (e.g., CART) and automated (e.g., Google Meet) services integrate speaker identity into captions, participants wanted it to be a universal feature. Many groups (A, B, C) favored conveying speaker identity in captioning by splitting captions up by speaker and using a visual indicator connected to their name (Table 5, A3). Groups initially proposed color-coding speakers, provided it met colorblindness and other visual accessibility standards, and later also considered using profile images. Group C envisioned other ways they might like to see speaker identity (by displaying captions under the active speaker's video, and with threads between the speaker and the captions), and therefore proposed providing multiple options that each user could select as desired. Separating captions by speaker and identifying the speaker's name were universally liked and showing threads between speakers and captions (as designed in our video prototype) was comprehensively disliked. Placing captions below the active speaker's video (Table 5, C3) garnered mixed reactions. Amelia liked that it would help consolidate information and limit instances where people “don't even look at the captions because [they're] too busy looking at the person talking” (Amelia). However, others felt that “would be a lot to follow” (Anna) and “take more cognitive effort” (Camille), particularly in meetings of five or more people.

Participants who frequently joined conversations with interpreters considered how speaker identity should work when an interpreter voices for multiple signers. They stressed that “when there are a lot of Deaf people it's not really entirely clear who's doing the actual uttering” (David) and that “there needs to be some way of the computer knowing to connect that interpreter to me so that [speaker identification] comes to me when I'm the one speaking” (Daisy). Deanna suggested enabling interpreters to click a button that would then “show who exactly they're voicing for” (Deanna).

Building on speaker identification, participants highlighted the importance of identifying overlapping speech for both DHH and hearing conversants. Overlapping speech poses a significant access barrier that is not well-addressed by current captioning solutions, leaving captioning users out of conversation. Barbara motivated the need for a tool to limit overlap by explaining; “[with] automatic captions, or even CART, there's no way to capture [overlap]... I'll just pull away emotionally or walk away physically.” Therefore, participants proposed both using speaker identification approaches to indicate overlap and sending conversation participants a pop-up when it occurs. When reacting to video prototypes of speaker overlap notifications (Table 5, B2), participants liked the baseline feature but had suggestions for nuances to build into future implementations. Participants stressed the importance of language choice in notifications, as having the pop up read ‘multiple speaker warning’ made participants feel like “something bad's gonna come out of the screen and grab you or something” (Barbara). Both displaying overlapping speakers’ names in the captioning and sending an alert was valuable to some—one hearing participant noted: “I would not have my captions on, so I do like that it pops up and makes it very, like, front and center” (Anna). When considering the impact of getting an alert, Blake remarked “[I liked] how annoying that is gonna [be] for teams that talk over each other a lot … because I think that creates an incentive … to create more accessible, inclusive conditions.” However, participants also envisioned possible negative impacts of penalizing overlapping speech: “people that are more shy, more self-conscious … may start to feel afraid to say anything” (Brian). Participants also worried that alerts could limit equitable turn-taking, as they could cause “the dilemma of ‘do I interrupt, or do I let this person take all the time?’” (Blake). Groups A and B liked that speaker overlap alerts could guide behavior change, since “it's something that happens over time with reminders” (Brenda), but others worried that that “most hearing people are used to being interrupted or talking over each other … so they don't want this visual alert” (Colin).

4.2.3 Support for Behavioral Feedback.

The next category of features involves a set of feedback mechanisms to alert the group to ways to make conversation more accessible. Participants proposed a variety of possible behaviors to alert conversation partners about, often using a similar pop-up implementation (Table 5, A2, B3, C1, D1, D3). Specifically, participants’ designs focused on providing captioning error feedback, asking others to adjust their camera, a communication breakdown alert, asking for a pause in conversation, and asking other speakers to slow down. While these access barriers have been discussed in prior work (e.g., [8,11,25,27,33,43,52,53]), we focus on how technical tools could help to mitigate them by guiding behavior change.

While Group A's design (Table 5, A2) included the opportunity to either flag or correct captioning errors in real time, participants only saw promise in flagging errors. Driving the focus on errors is the reality that, while DHH participants recounted many nonsensical errors in captions, such as “calling ‘site administrator’ ‘satan administrator’” (Bea), hearing conversation partners rarely noticed or tried to address them. Despite “lov[ing] the idea of being able to correct [captions] as we go” (Daisy), participants concluded that doing so was “a little too far-fetched” (Anna) because it would be cognitively overwhelming and likely too delayed to be useful. However, participants saw social benefit in being able to flag caption errors anonymously, imagining that it would help “that shy person who doesn't want to interrupt” (Blake) and allow users to ask for repeated clarification without feeling like there's “a target on you” (Amelia). However, others worried that it could make “the flow of the conversation stop” (Dot). While the video prototype simply alerted that an error occurred, participants proposed that the alert should point out the confusing caption in context, since it was likely said “10 seconds ago, and then you're like, ‘Oh, well, what word was it?’” (Colin). In summary, real-time error correction may not be feasible, but participants were enthusiastic about being able to call their conversation partners’ attention to errors that impacted comprehension.

For participants who relied on seeing conversation partners clearly, being able to discreetly ask someone to adjust their camera view was exciting, but discussion revealed social complexity in doing so. Many participants shared Amelia's video conferencing experience of feeling “like, I don't know who's talking … can you please turn on your camera.” Participants saw the use case for being able to alert someone to adjust their camera – in fact, while discussing this video prototype (Table 5, B3), Daisy took the opportunity to tell David “you're kind of cut off at the neck … you gotta move.” However, participants noted that just telling someone to adjust their camera without a reason or specific directions was too ambiguous, proposing that the alert could specifically mention “someone can't read your lips right now” (Blake). However, participants also pointed out reasons why camera use was not always desired. For some “the exhaustion of being on camera all the time” (Bridget) is significant, and others found that only having active speakers or signers on camera could help minimize visual overload (Barbara). Participants also considered innocuous reasons why someone might not be visible on camera, such as when “someone could be holding their puppy and it's in front of their face and you can't see their face or lips” (Blake). Participants stressed the importance of thinking about the need for clear camera feeds within broader social context and cautioned that norms around using such an alert needed to account for nuance.

Participants also spoke to the need for a mechanism to tell conversation partners to slow down and were positive about Group D's design (Table 5, D3). Speech rate alerts were considered in context of participants’ current strategies to get speakers to slow down. When considering how groups might address speech rate, Brian shared how he approaches new vendors at work who “just talk too fast.” He begins conversations by saying “’I would really appreciate it if you just slow down your voice, just a bit, so I can follow what you're saying.’ … They will at some point speed up again; [I] just kindly remind them.” When assessing the speech rate video prototype, Amelia imagined that it would be “a good way to teach people how to have good speaking skills” by providing a mechanism to unobtrusively remind speakers when they speed up. Blake saw additional benefit in being able to get feedback on her speaking rate, saying, “having hearing loss … we don't really have to hear ourselves, I'm always someone who … tries to work on slowing down”. While the prototype showed manual speech rate alerts, participants also proposed automated speech rate monitoring, either in the form of auto-generated alerts (Brian) or adding a speedometer visualization for speakers (Daisy, Dot).

The final two behavioral feedback ideas, a pause (Table 5, D1) and attention (Table 5, D3) button, had similar motivations – identifying and encouraging groups to address conversational breakdown–and participants had mixed reactions to both. Participants from Groups C and D, who frequently communicate via interpreters, proposed a way to address the fact that delays in communication often mean that “once I finally get to that point where I can actually add something [to the discussion] … now it's the wrong time” (Daisy). However, despite agreeing that this was a common problem that needs addressing, participants reactions to attention or pause pop up alerts focused on the need for stronger guidance and mixed feelings around halting conversations. Anna suggested that alerts provide more specific guidance than simply calling for ‘attention’ as she felt that notification would cause her to “panic and … not know what to do from there.” Most participants worried that a pause or attention button would be too disruptive to a conversation or become “something that gets abused” (Amelia). However, Blake “got really excited” about building tools into a system that could “empower the person who's maybe too timid to speak.” Participants brainstormed ways to address the need to identify breakdown while minimizing disruption, proposing that it could be “up to the person who's pressing the alert button whether or not they want to send that alert just to the host, or to everyone” (Colin). Participants were united on the importance of calling attention to conversation breakdowns but after watching their simulated use, concluded that disruptive alerts were not the right tool to address this need.

4.2.4 Videoconferencing Infrastructure for Accessibility.

Another target for technology that could support groups in more accessible conversation was videoconferencing platform infrastructure itself. Participants focused on adding a new set of access norms in software settings that could build group norms into the platform (Table 5, B1) and desired greater customizability over current captioning interfaces (Table 5, C2). Customizability has been highlighted throughout prior work [11,12,18,43], but the role of platform infrastructure on accessibility for DHH people has so far only been explored in the context of sign language use [50].

Group B's access profile (Table 5, B1) allowed groups to enable desired features (such as the behavioral feedback tools discussed in Section 4.2.2) and share social norms. Across groups, participants were excited about the idea of an access profile and brainstormed ways to address the many complexities it introduces. Camille reflected that “we want technology to solve things, but we realize that people have to modify, they have to change” and imagined that building access norms into a system was a way to “leverage technology to help.” Participants highlighted the benefit of having preset but highly configurable options, as there are common issues that “d/Deaf people agree are the pain points when attending an online meeting,” but for individual groups, settings “should be able to be customizable” (Colin). Additionally, Blake considered how, often when joining a meeting, “I'm going really fast and I'm not setting things up ahead of time” and Barbara suggested that settings “should be real easy for the consumer to turn on or off, even if you're in the middle of a Zoom meeting.” Bea proposed that users should be able to save context-specific presets, for “big meetings, classes, small meetings.” While participants valued having an anonymous way to request their needed access supports and norms, questions arose around misuse (e.g., malicious users). Broadly, participants were excited about the possibility to build accessible conversation norms into videoconferencing systems and continued to think through the nuanced factors that would make such a tool effective.

Highly customizable captioning displays were an additional area of videoconferencing infrastructure with unified support. Customization is not a new concept – in fact Amelia responded to this prototype (Table 5, C2) by “check[ing] the subtitle settings. I was, like, ‘do they not have settings for captions?’ And they don't. I was really surprised.” In light of the lack of control in current tools, participants highlighted the specific dimensions that were important to them. Customizable colors were critical, as Cad explained that for “DeafBlind people who have some vision but need some accommodation” there can be some “contrast of colors [or] particular colors that are better than others.” Other features included resizing the “short little box” (Camille) used to display captions and supporting users’ preferred setups by letting “captions show up in a separate browser tab” (Colin). While not necessarily a novel technological innovation, the control that platforms do or don't allow their users significantly shapes who can use captions effectively, and participants highlighted the need for greater control and customizability.

4.2.5 Sound Information.

In the final category of designs, participants proposed providing more information about sound in addition to transcription. Designs included visualizing speaker's volume (Table 5, A1) and identifying non-speech sounds (Table 5, D2). Though Google has integrated sound recognition into the Live Transcribe app [69], it is not available within commercial videoconferencing tools. Volume visualization has only been explored in the context of pre-recorded captions [23].

Group A's proposal to visualize speaker volume in captions (Table 5, A1) seemed promising during their brainstorming but, in viewing it, participants identified more problems than benefits. Amelia initially was motivated to display volume as a proxy for tone, which is “very easily misunderstood just reading the captions” (Amelia). However, many questioned if simply showing volume could lead to misunderstandings. For instance, knowing that a person is speaking quietly could pose the question: are they “unsure or meek? Or is it just that they [a] quiet talker?” (Colin). Additionally, participants considered that volume may not always vary significantly, such as with Audrey's family “who have one volume, and it's yelling … it would just be bold the whole time.” While participants liked the idea of volume displays that were “dynamic without being disruptive” (Daisy), they concluded that this implementation would not be useful.

In addition to captions, participants were interested in identifying and displaying background noise during a conversation. Daisy described why background noise notification is needed: “I can't tell you how many times I'll be talking to someone … and the hearing person suddenly looks off into the horizon … I'm like ‘hey, what's going on? Why, you know, why is the conversation being disrupted? I can't hear that’.” DHH participants valued the idea of displaying background noise within a videoconferencing tool with both an emoji and text description of the sound (Table 5, D2), though hearing people questioned its necessity. Participants favored using emojis paired with background noise descriptions to “just get a quick bit of information” (Daisy), and the colorful nature of emojis makes alerts “really bright and easy to capture” (Deanna). However, rather than placing background noise alerts in video feeds, participants suggested they would be “better in a bottom corner” (Dot) or “between sentences in the captioning” (Deanna) to avoid splitting users’ visual attention. When considering the social implications of this design, some hearing participants believed that background noise happening around them is “not necessary for other people to know this, it's really more for the speaker” (Bridget). However, DHH participants stressed that knowing a noise is happening “just gives us clarification of why you're pausing” (Amelia), and that it would be useful for people with some residual hearing to know “is [a noise] me or is it someone else?” (Barbara). Participants also wondered how to determine “what the threshold is” (Brenda) for identifying sounds–as Colin put it, “some hearing people hear background noise and either intentionally or unintentionally ignore it … and other times it's like ‘whoa I heard that fire alarm’.” Being aware of background noise is critical for DHH conversants, and participants brainstormed how to best communicate that over videoconferencing platforms.

4.2.6 Summary.

Our participants first designed features to address their most salient communication hurdles and then, by engaging with video prototypes, surfaced aspects of their designs that would need to be carefully considered to fit the social norms of mixed-hearing ability group conversations. Participants identified features that have already been implemented in captioning in the past, namely speaker identification and caption customization, as high-priority and high-impact to universally build into videoconferencing tools. Guiding groups to be more aware of speaker overlap, speech rate, comprehension-critical caption errors, and the need to adjust their camera via pop-up notifications are promising features for future development and innovation. Participants also wanted to be able to set, share, and customize access practices within videoconferencing platforms, and this is a promising new paradigm. Finally, products like Live Transcribe [69] have integrated sound recognition into their ASR apps, and our findings indicate that this would be valuable during videoconferences as well. Participants also identified features they did not want. While increasing group awareness of conversation breakdowns was important, pop-up alerts did not prove to be an appropriate approach. Additionally, displaying raw volume information by styling the captions themselves was seen as distracting and unclear.