Nov 2021 (4 weeks)
DhanMitra
DhanMitra
“Inclusion means creating equal
opportunities, not just accommodations.”
Gesture Detection
Text-to-speech
Nov 2021 (4 weeks)
DhanMitra
“Inclusion means creating equal
opportunities, not just accommodations.”
Gesture Detection
Text-to-speech
Context
Context
Academic Project
Academic Project
Academic Project
Role
Role
Concept Generation & Initial Prototyping
Concept Generation & Initial Prototyping
Concept Generation & Initial Prototyping
Collaboration (Team)
Collaboration (Team)
Sahana Sundar, Ranveer Singh
Sahana Sundar, Ranveer Singh
Sahana Sundar, Ranveer Singh
What is DhanMitra?
What is DhanMitra?
DhanMitra is a digital platform that simplifies loan processes in Tier 2 and Tier 3 cities by connecting local borrowers with trusted agents and banks.
(The Design Process)
The Current Experience
The Current Experience
Analyzing accessibility features on Video Calls
Analyzing accessibility features on Video Calls
User Research
User Research
Analyzing existing sign language communication
solutions to identify Design Opportunities.
Analyzing existing sign language communication
solutions to identify Design Opportunities.
Design Goals
Sign language users need an accessibility feature that allows them to:
(Solutioning)
That “AH-HA” Moment
That “AH-HA” Moment
The technology behind the idea already existed, but no one had combined the two, "hand detection and voice output" and applied them in a video calling context. We saw this gap and seized the opportunity to bridge it.
The technology behind the idea already existed, but no one had combined the two, "hand detection and voice output" and applied them in a video calling context. We saw this gap and seized the opportunity to bridge it.
How does it solve the problem?
How does it solve the problem?
The solution aims to make communication equal, not just an accommodation.
Gesture Detection
Gesture Detection
The camera on the sign language user’s device captures their gestures, allowing them to communicate independently without relying on interpreters.
The camera on the sign language user’s device captures their gestures, allowing them to communicate independently without relying on interpreters.
Seamless Translation
Seamless Translation
Gestures are converted into text and then output as speech for other participants, so others can follow the conversation without needing to focus on captions or adjust their comprehension. This gives sign language users a chance at equal communication.
Gestures are converted into text and then output as speech for other participants, so others can follow the conversation without needing to focus on captions or adjust their comprehension. This gives sign language users a chance at equal communication.
Reduced Visibility Pressure
Reduced Visibility Pressure
Users can turn off their video while still being heard, removing the pressure of constant visibility while ensuring full participation in the conversation.
Users can turn off their video while still being heard, removing the pressure of constant visibility while ensuring full participation in the conversation.
(Initial Prototyping)
Experimenting with AI & ML
1. Hand Detection Model
1. Hand Detection Model
An initial exploration with gesture detection using the Fingerpose library, assigning actions to gestures.
An initial exploration with gesture detection using the Fingerpose library, assigning actions to gestures.
Coded using: Tensorflow & React.js
2. Double Hand Detection
2. Double Hand Detection
Expanded the experiment to detect specific poses and signs with double-hand detection.
Expanded the experiment to detect specific poses and signs with double-hand detection.
Coded using: Mediapipe and open CV
System Architecture
System Architecture
P.S. While this would require collaboration with tech experts to figure out the details, we’ve explored the technology and systems and outlined potential implementation suggestions based on our understanding.
P.S. While this would require collaboration with tech experts to figure out the details, we’ve explored the technology and systems and outlined potential implementation suggestions based on our understanding.
Latest Update: July 2024
Latest Update: July 2024
Apple recently made the first limitation possible with their latest update ! It detects gesture, plays an animation coverts it into one video and send it over to through video calls all in real-time!
Apple recently made the first limitation possible with their latest update ! It detects gesture, plays an animation coverts it into one video and send it over to through video calls all in real-time!
The Final Prototype
Make sure to watch the prototype video with sound for the full experience! 🎥🔊
The Imapct
I posted a video of the project on LinkedIn and it went viral!
I posted a video of the project on LinkedIn and it went viral!
Sparked a conversation among 70 thousand people worldwide around accessibility in emerging technologies, raising awareness of the possibilities of accessible design with AI/ML.
Connected with 20+ hard-of-hearing and speech-impaired user, receiving direct feedback and learning firsthand the value of initiatives in accessible design.
Reached over 4 Million people globally including industry leaders, designers, non-designers, and accessibility advocates, making it one of my most visible and widely received projects.
Sparked a conversation among 70 thousand people worldwide around accessibility in emerging technologies, raising awareness of the possibilities of accessible design with AI/ML.
Connected with 20+ hard-of-hearing and speech-impaired user, receiving direct feedback and learning firsthand the value of initiatives in accessible design.
Reached over 4 Million people globally including industry leaders, designers, non-designers, and accessibility advocates, making it one of my most visible and widely received projects.
Sparked a conversation among 70 thousand people worldwide around accessibility in emerging technologies, raising awareness of the possibilities of accessible design with AI/ML.
Connected with 20+ hard-of-hearing and speech-impaired user, receiving direct feedback and learning firsthand the value of initiatives in accessible design.
Reached over 4 Million people globally including industry leaders, designers, non-designers, and accessibility advocates, making it one of my most visible and widely received projects.
