News

31 Jan 2023

Smart walking stick in development to help blind shoppers navigate supermarkets and cafes

Bettina Badon

US engineers are developing a smart walking stick for blind and visually impaired people using advances in artificial intelligence.

The smart walking stick could help blind people navigate tasks from shopping for a bag of rice at the supermarket to choosing a quiet place to sit in a busy cafe. It comes from a study at the University of Colorado Boulder by doctoral student Shivendra Agrawal and his colleagues of the collaborative AI and robotics lab.

“I really enjoy grocery shopping and spend a significant amount of time in the store,” Mr Agrawal comments. “A lot of people can’t do that however, and it can be really restrictive. We think this is a solvable problem.”

In his study published in October, Mr Agrawal and his fellow researchers got one step closer to finding a solution. The team are developing a smart walking stick which resembles the white and red canes available to purchase at Walmart. But it also includes a few additional features such as a camera and computer vision technology to map and record the world around it.

Using vibrations in its handle, the gadget then guides users with the help of spoken instructions like ‘reach a little bit to your right.’ Mr Agrawal says that while the device is not supposed to be a substitute for designing places like supermarkets, he hopes the prototype will show how AI can help millions of people to become more independent.

“AI and computer vision are improving, and people are using them to build self-driving cars and similar inventions,” he explains. “But these technologies also have the potential to improve quality of life for many people.”

The researcher and his team first explored the potential of a smart walking stick by tackling the common question ‘where do I sit?’

He adds: “Imagine you’re in a café. You don’t want to sit just anywhere. You usually take a seat close to the walls to preserve your privacy, and you usually don’t like to sit face-to-face with a stranger.”

Smart walking stick research

Credit: University of Colorado Boulder

Existing research suggests that people who are blind or visually impaired tend to prioritise these type of decisions. To see if their smart walking stick could help, the researchers set up a cafe of sorts in their lab, complete with chairs, customers, and obstacles.

Participants of the study strapped on a backpack with a laptop inside and then picked up the smart walking stick, turning to survey the room with the camera attached near its handle. Similar to a self-driving car, algorithms running inside the laptop identified the various features in the room and calculated the route to an ideal seat.

The computer vision algorithm can also rank boxes to identify the product the user wants, such a particular box of cereals. The researchers shared their findings at the International Conference on Intelligent Robots and Systems in Kyoto, Japan. Other team members include Bradley Hayes, assistant professor of computer science, and doctoral student Mary Etta West.

“Shivendra’s work is the perfect combination of technical innovation and impactful application, going beyond navigation to bring advancements in under-explored areas, such as assisting people with visual impairment with social convention adherence or finding and grasping objects,” Hayes said.

In new research, not yet published, Mr Agrawal and his colleagues adapted their device for a task that can be daunting for anyone – finding and grasping products in aisles filled with dozens of similar-looking and similar-feeling choices.

Again, the team set up a makeshift environment in their lab: this time, a grocery shelf stocked with several different kinds of cereal. The researchers created a database of product photos, such as boxes of Honey Nut Cheerios or Apple Jacks, into their software. Study subjects then used the smart walking stick to scan the shelf, searching for the product they wanted.

“It assigns a score to the objects present, selecting what is the most likely product,” Mr Agrawal said. “Then the system issues commands like ‘move a little bit to your left.’”

He added that it will be a while before the team’s walking stick makes it into the hands of real shoppers. The group, for example, wants to make the system more compact, designing it so it can run off a standard smartphone attached to a cane. 

But the researchers also hope their preliminary results for the smart walking stick will inspire other engineers to rethink what robotics and AI are capable of.

Loading