DanicaMartins

DanicaMartins

DanicaMartins

Nov–Dec 2021

Signify

Designed an inclusive video calling experience addressing accessibility gaps, reaching 4Million+ people globally and supporting a community of 40+ sign language users.

Designed an inclusive video calling experience addressing accessibility gaps, reaching 4Million+ people globally and supporting a community of 40+ sign language users.

Gen-AI

ML

InclusiveTechnology

Innovation

Nov–Dec 2021

Signify

Sahana Sundar (Carnegie Mellon University)

Ranveer Singh (TenXer Labs)Designed an inclusive video calling experience addressing accessibility gaps, reaching 4Million+ people globally and supporting a community of 40+ sign language users.

Gen-AI

ML

InclusiveTechnology

Innovation

A group of four people engaged in a discussion while seated in a cozy, plant-filled setting.

A group of four people engaged in a discussion while seated in a cozy, plant-filled setting.

A group of four people engaged in a discussion while seated in a cozy, plant-filled setting.

Role

Role

Me (Ideation, Design, Initial Prototyping)

Me (Ideation, Design, Initial Prototyping)

Collaboration (Team)

Collaboration (Team)

Sahana Sundar (Research, Design)

Ranveer Singh (Design, Final Prototyping)

Sahana Sundar (Research, Design)

Ranveer Singh (Design, Final Prototyping)

The Problem

The Problem

Accessibility features on video calling platforms often feel like accommodations, not equal opportunities. For sign language users, this means more barriers that highlight differences.

Accessibility features on video calling platforms often feel like accommodations, not equal opportunities. For sign language users, this means more barriers that highlight differences.

(The Design Process)

The Current Experience

The Current Experience

Analyzing accessibility features on Video Calls

Analyzing accessibility features on Video Calls

User Research

User Research

Conducted 1:1 interviews combined with usability testing sessions with 2 participants from the hard-of-hearing and speech-impaired community.

Secondary Research

Secondary Research

Analyzing existing sign language communication
solutions to identify Design Opportunities.

Analyzing existing sign language communication
solutions to identify Design Opportunities.

Design Goals

Sign language users need an accessibility feature that allows them to:

(Solutioning)

That “AH-HA” Moment

That “AH-HA” Moment

The technology behind the idea already existed, but no one had combined the two, "hand detection and voice output" and applied them in a video calling context. We saw this gap and seized the opportunity to bridge it.

The technology behind the idea already existed, but no one had combined the two, "hand detection and voice output" and applied them in a video calling context. We saw this gap and seized the opportunity to bridge it.

(Initial Prototyping & Testing)

Experimenting with AI & ML

1. Hand Detection Model

1. Hand Detection Model

An initial exploration with gesture detection using the Fingerpose library, assigning actions to gestures.

An initial exploration with gesture detection using the Fingerpose library, assigning actions to gestures.

Coded using: Tensorflow & React.js

Tutorial followed for Hand Detection and Gesture Recognition

2. Double Hand Detection

2. Double Hand Detection

Expanded the experiment to detect specific poses and signs with double-hand detection.

Expanded the experiment to detect specific poses and signs with double-hand detection.

Coded using: Mediapipe and open CV

Tutorial followed for Hand Detection and Gesture Recognition

(Testing Results)

How does it solve the problem?

How does it solve the problem?

The solution aims to make communication equal, not just an accommodation.

Gesture Detection

Gesture Detection

Real-time gesture capture lets sign language users communicate naturally in ISL, without the need for interpreters.

Seamless Translation

Seamless Translation

Gestures are converted into text and then output as speech for other participants, so others can follow the conversation without needing to read captions.

Reduced Visibility Pressure

Reduced Visibility Pressure

Users can turn off their video while still being heard, removing the pressure of constant visibility.

Users can turn off their video while still being heard, removing the pressure of constant visibility.

System Architecture

System Architecture

P.S. While this would require collaboration with tech experts to figure out the details, we’ve explored the technology and systems and outlined potential implementation suggestions based on our understanding.

P.S. While this would require collaboration with tech experts to figure out the details, we’ve explored the technology and systems and outlined potential implementation suggestions based on our understanding.

Latest Update: July 2024  

Latest Update: July 2024  

Apple recently made the first limitation possible with their latest update !  It detects gesture, plays an animation coverts it into one video and send it over to through video calls all in real-time!

Apple recently made the first limitation possible with their latest update !  It detects gesture, plays an animation coverts it into one video and send it over to through video calls all in real-time!

The Final Prototype

Make sure to watch the prototype video with sound for the full experience! 🎥🔊

The Imapct

I posted a video of the project on LinkedIn and it went viral!

I posted a video of the project on LinkedIn and it went viral!

View LinkedIn Post

View LinkedIn Post

View LinkedIn Post

View LinkedIn Post

View LinkedIn Post

View LinkedIn Post

Sparked a conversation among 70 thousand people worldwide around accessibility in emerging technologies, raising awareness of the possibilities of accessible design with AI/ML.
Connected with 40+ hard-of-hearing and speech-impaired user, receiving direct feedback and learning firsthand the value of initiatives in accessible design.
Reached over 4 Million people globally including industry leaders, designers, non-designers, and accessibility advocates, making it one of my most visible and widely received projects.

Sparked a conversation among 70 thousand people worldwide around accessibility in emerging technologies, raising awareness of the possibilities of accessible design with AI/ML.
Connected with 40+ hard-of-hearing and speech-impaired user, receiving direct feedback and learning firsthand the value of initiatives in accessible design.
Reached over 4 Million people globally including industry leaders, designers, non-designers, and accessibility advocates, making it one of my most visible and widely received projects.

Sparked a conversation among 70 thousand people worldwide around accessibility in emerging technologies, raising awareness of the possibilities of accessible design with AI/ML.
Connected with 40+ hard-of-hearing and speech-impaired user, receiving direct feedback and learning firsthand the value of initiatives in accessible design.
Reached over 4 Million people globally including industry leaders, designers, non-designers, and accessibility advocates, making it one of my most visible and widely received projects.

For more details about the project, I’d be happy to connect with you!