An AI-driven voice user interface with head-mounted camera for the blind to improve their object and text recognition process.

Capstone Project Sponsored by Amazon

Tools
Interaction Design
System Design
Voice UI Design

TimelineMar - Aug 2020

Team
Giada Sun
Yagmur Erten


Product Demo Video scripted and co-directed by Weixi


According to CDC, there are around 4.3 million people in the US with blindness and visual impairment. They often face daily obstacles to perceive their environments in a world that mostly is designed BY and FOR the sighted people, such as reading printed instructions, sorting mails, and differentiating medicine bottles.



Usability Issues with Existing Technology


Current computer vision technologies such as Seeing AI that aim to help people with blindness in text and object recognition come with usability issues, such as requiring users to aim the camera on an object directly and to frame a clear picture for successful recognition, most of them also require user to use one or both hands to frame the picture while holding an object.

00
Context

How might we assist the blind in retrieving relevant product information through an independent experience?

01
Challenge

Interaction Design
Designed the conversational experience of the smart glasses, building sample dialogues and VUI flow,

Prototype Testing
Design the prototype and testing sessions to iteratively improve the experience leveraging user's existing behaviors.

Demo Video
Scripted and directed a video to showcase the features and scenarios when IRIS can bring value to the users.

User Research
Designed research to understand and design with users who are blind, understanding current assistive technology usage and shopping behavior. Synthesized insights, design principles and design opportunities.

02
My Contribution
03
What is Iris
04
How it works

Simply hold an object
Iris aims to remove the restrictions that current technologies have by allowing users to recognize objects through simply holding or pointing at an item with their hands.

Ask for desired information
Users can ask Iris for specific information from the item that they are holding. For example, the ingredient of the sanitizer gel.


Receive information directly
Iris’ sound system and haptic feedback allow users to ask and receive the information through headphones if they need privacy.

05
Expert Interviews

With the information we gathered from secondary research and articles, we gained a general understanding of how blind people use the Internet. With that knowledge, we spoke with experts in this field to expand our understanding of blind people’s online shopping behaviors.

The key takeaways from those interviews are:

06
Competitive Assessment
Shopping Process

How do blind people navigate on online shopping platforms, compare products, and return items?
What kind of products do blind people tend to purchase online vs in-store?
How do blind people assess shopping platforms’ accessibility?

Support & Tools

What tools are available/missing for blind people while shopping online?
How can we support blind people while they shop online?

06
Research
Methods

9 Phone Interviews with Directed Storytelling
The goal was to understand our user's general shopping journey by collecting first-hand data of blind people’s past experiences, opinions, and attitudes around online shopping. We also wanted to identify any advancements and frustrations that exist within the current technologies


5 Remote Moderated Studies
We asked participants to share their screens with us through Zoom, and requested them to do online shopping exercises as we observed their patterns. We watched our participants' steps as they browsed and selected both familiar and unfamiliar items. Our goal was to gain a deep understanding of blind people’s online shopping behaviors and the tools that use and pinpoint any missing features that could ease their process of online shopping.

07
Analysis
& Insights



Intentional Shopping For Efficiency
Most blind people are intentional shoppers, who are not interested in browsing items without purpose or taking website’s recommendations as reference.




Familiarity and Recurring Items
Familiarity with items and websites during shopping is crucially important to blind customers.




Relevant and Specific Information
Blind users are specific about relevant information about products: they use screen readers to skim websites through only reading key elements.




Inter-dependent with Assistive Technology

Most blind people like using VUI devices for short commands to complete every
day tasks, such as setting alarms, checking time and the weather.

05
Research
Questions
08
Design principles


09
User Needs


10
Ideation & 
Down Selection


11
Storyboard


12
Prototype
& Usability Tests

More design iterations and specifications coming soon ... Stay tuned!




Galene
🌊  App & System Design
Star Travel
Product & Website Design  👁️

Designed & Illustrated by Weixi Zhang

emaillinkedin