Signify
“Inclusion means creating equal
opportunities, not just accommodations.”
Signify
“Inclusion means creating equal
opportunities, not just accommodations.”
Role
Role
Lead Designer
Lead Designer
Collaboration (Team)
Collaboration (Team)
Sahana Sundar (Carnegie Mellon University)
Ranveer Singh (TenXer Labs)
Sahana Sundar (Carnegie Mellon University)
Ranveer Singh (TenXer Labs)
The Problem
The Problem
Accessibility features on video calling platforms often feel like accommodations, not equal opportunities. For sign language users, this means more barriers that highlight differences.
Accessibility features on video calling platforms often feel like accommodations, not equal opportunities. For sign language users, this means more barriers that highlight differences.
(The Design Process)
The Current Experience
The Current Experience
Analyzing accessibility features on Video Calls
Analyzing accessibility features on Video Calls
User Research
User Research
Conducted 1:1 interviews combined with usability testing sessions with 2 participants from the hard-of-hearing and speech-impaired community.
Secondary Research
Secondary Research
Analyzing existing sign language communication
solutions to identify Design Opportunities.
Analyzing existing sign language communication
solutions to identify Design Opportunities.
Design Goals
Sign language users need an accessibility feature that allows them to:
(Solutioning)
That “AH-HA” Moment
That “AH-HA” Moment
The technology behind the idea already existed, but no one had combined the two, "hand detection and voice output" and applied them in a video calling context. We saw this gap and seized the opportunity to bridge it.
The technology behind the idea already existed, but no one had combined the two, "hand detection and voice output" and applied them in a video calling context. We saw this gap and seized the opportunity to bridge it.
(Initial Prototyping & Testing)
Experimenting with AI & ML
1. Hand Detection Model
1. Hand Detection Model
An initial exploration with gesture detection using the Fingerpose library, assigning actions to gestures.
An initial exploration with gesture detection using the Fingerpose library, assigning actions to gestures.
Coded using: Tensorflow & React.js
2. Double Hand Detection
2. Double Hand Detection
Expanded the experiment to detect specific poses and signs with double-hand detection.
Expanded the experiment to detect specific poses and signs with double-hand detection.
Coded using: Mediapipe and open CV
(Testing Results)
How does it solve the problem?
How does it solve the problem?
The solution aims to make communication equal, not just an accommodation.
Gesture Detection
Gesture Detection
The camera on the sign language user’s device captures their gestures, allowing them to communicate independently without relying on interpreters.
The camera on the sign language user’s device captures their gestures, allowing them to communicate independently without relying on interpreters.
Seamless Translation
Seamless Translation
Gestures are converted into text and then output as speech for other participants, so others can follow the conversation without needing to focus on captions or adjust their comprehension. This gives sign language users a chance at equal communication.
Gestures are converted into text and then output as speech for other participants, so others can follow the conversation without needing to focus on captions or adjust their comprehension. This gives sign language users a chance at equal communication.
Reduced Visibility Pressure
Reduced Visibility Pressure
Users can turn off their video while still being heard, removing the pressure of constant visibility while ensuring full participation in the conversation.
Users can turn off their video while still being heard, removing the pressure of constant visibility while ensuring full participation in the conversation.
System Architecture
System Architecture
P.S. While this would require collaboration with tech experts to figure out the details, we’ve explored the technology and systems and outlined potential implementation suggestions based on our understanding.
P.S. While this would require collaboration with tech experts to figure out the details, we’ve explored the technology and systems and outlined potential implementation suggestions based on our understanding.
Latest Update: July 2024
Latest Update: July 2024
Apple recently made the first limitation possible with their latest update ! It detects gesture, plays an animation coverts it into one video and send it over to through video calls all in real-time!
Apple recently made the first limitation possible with their latest update ! It detects gesture, plays an animation coverts it into one video and send it over to through video calls all in real-time!
The Final Prototype
Make sure to watch the prototype video with sound for the full experience! 🎥🔊
The Imapct
I posted a video of the project on LinkedIn and it went viral!
I posted a video of the project on LinkedIn and it went viral!
Sparked a conversation among 70 thousand people worldwide around accessibility in emerging technologies, raising awareness of the possibilities of accessible design with AI/ML.
Connected with 20+ hard-of-hearing and speech-impaired user, receiving direct feedback and learning firsthand the value of initiatives in accessible design.
Reached over 4 Million people globally including industry leaders, designers, non-designers, and accessibility advocates, making it one of my most visible and widely received projects.
Sparked a conversation among 70 thousand people worldwide around accessibility in emerging technologies, raising awareness of the possibilities of accessible design with AI/ML.
Connected with 20+ hard-of-hearing and speech-impaired user, receiving direct feedback and learning firsthand the value of initiatives in accessible design.
Reached over 4 Million people globally including industry leaders, designers, non-designers, and accessibility advocates, making it one of my most visible and widely received projects.
Sparked a conversation among 70 thousand people worldwide around accessibility in emerging technologies, raising awareness of the possibilities of accessible design with AI/ML.
Connected with 20+ hard-of-hearing and speech-impaired user, receiving direct feedback and learning firsthand the value of initiatives in accessible design.
Reached over 4 Million people globally including industry leaders, designers, non-designers, and accessibility advocates, making it one of my most visible and widely received projects.
