Assistive system for people with visual impairments

Patent metadata

Field	Value
Publication number	US-12505657-B2
Application number	US-202217582714-A
Country	US
Kind code	B2
Filing date	Jan 24, 2022
Priority date	Jan 25, 2021
Publication date	Dec 23, 2025
Grant date	Dec 23, 2025

Abstract

Systems, methods, apparatuses, and computer program products for assisting people with visual impairments. A method for operating a mobile assistive device may be provided. The method may include capturing an image of a real-time scene of an environment. The method may also include sending the image to a single-board computer. The method may further include processing the image. In addition, the method may include providing navigation assistance to a user based on the processed image.

First claim

We claim:

1 . A method for operating a mobile assistive device, comprising:
capturing an image of a real-time scene of an environment; sending the image to a single-board computer; processing the image; providing navigation assistance to a user based on the processed image, wherein the processing of the image comprises at least one of recognizing a location of the user in the environment, recognizing or detecting an object in the environment, detecting static or dynamic obstacles in the environment, and performing an end-to-end text recognition of the image, wherein performance of the end-to-end text recognition comprises
implementing text detection and text recognition, and
implementing a recurrent neural network by stacking convolutional, pooling and dense layers with different filter sizes and different number of filters; and
controlling the recognition or detection of the object, and the performance of the end-to-end text recognition of the image via verbal instructions from the user, wherein the controlling comprises turning on or off, and pausing or resuming the object recognition or detection and the end-to-end text recognition of the image.

2 . The method for operating the mobile assistive device according to claim 1 , wherein recognition of the location of the user in the environment comprises utilizing a lite convolutional neural network model configured to recognize the environment.

3 . The method for operating the mobile assistive device according to claim 1 , wherein recognition or detection of the of the object in the environment comprises employing machine-learning trained to identify objects and obstacles found in the environment.

4 . The method for operating the mobile assistive device according to claim 1 , wherein detection of the static or dynamic obstacles in the environment comprises using a micro light detection and ranging method.

5 . The method for operating the mobile assistive device according to claim 1 , wherein the navigation assistance is provided via speech instructions.

6 . A mobile assistive device, comprising:
at least one processor; and at least one memory including computer program code, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause a controller of the mobile assistive device at least to: capture an image of a real-time scene of an environment; send the image to a single-board computer; process the image; provide navigation assistance to a user based on the processed image, wherein processing the image comprises at least one of recognizing a location of the user in the environment, recognizing or detecting an object in the environment, detecting static or dynamic obstacles in the environment, and performing an end-to-end text recognition of the image, wherein performance of the end-to-end text recognition comprises
implementing text detection and text recognition, and
implementing a recurrent neural network by stacking convolutional, pooling and dense layers with different filter sizes and different number of filters; and
control the recognition or detection of the object, and the performance of the end-to-end text recognition of the image via verbal instructions from the user, wherein the control comprises turning on or off, and pausing or resuming the object recognition or detection and the end-to-end text recognition of the image.

7 . The mobile assistive device according to claim 6 , wherein when recognizing the location of the user in the environment, the at least one memory and the computer program code are further configured to, with the at least one processor, cause the controller at least to:
utilize a lite convolutional neural network model configured to recognize the environment.

8 . The mobile assistive device according to claim 6 , wherein when recognizing or detecting the object in the environment, the at least one memory and the computer program code are further configured to, with the at least one processor, cause the controller at least to:
employ machine-learning trained to identify objects and obstacles found in the environment.

9 . The mobile assistive device according to claim 6 , wherein when detecting the static or dynamic obstacles in the environment, the at least one memory and the computer program code are further configured to, with the at least one processor, cause the controller at least to:
execute a micro light detection and ranging method.

10 . The mobile assistive device according to claim 6 , wherein the navigation assistance is provided via speech instructions.

11 . A non-transitory computer readable medium, with a computer program encoded thereon, the computer program, when executed by a processor, causes the processor to:
capture an image of a real-time scene of an environment; send the image to a single-board computer; process the image; provide navigation assistance to a user based on the processed image, wherein processing the image comprises at least one of recognizing a location of the user in the environment, recognizing or detecting an object in the environment, detecting static or dynamic obstacles in the environment, and performing an end-to-end text recognition of the image, wherein performance of the end-to-end text recognition comprises
implementing text detection and text recognition, and
implementing a recurrent neural network by stacking convolutional, pooling and dense layers with different filter sizes and different number of filters; and
control the recognition or detection of the object, and the performance of the end-to-end text recognition of the image via verbal instructions from the user, wherein the control comprises turning on or off, and pausing or resuming the object recognition or detection and the end-to-end text recognition of the image.

12 . The non-transitory computer readable medium according to claim 11 , wherein when recognizing the location of the user in the environment, the processor is further caused to:
utilize a lite convolutional neural network model configured to recognize the environment.

13 . The non-transitory computer readable medium according to claim 11 , wherein when recognizing or detecting the object in the environment, the processor is further caused to:
employ machine-learning trained to identify objects and obstacles found in the environment.

14 . The non-transitory computer readable medium according to claim 11 , wherein when detecting the static or dynamic obstacles in the environment, the processor is further caused to:
execute a micro light detection and ranging method.

15 . The non-transitory computer readable medium according to claim 11 , wherein the navigation assistance is provided via speech instructions.

Assignees

• Univ Qatar

Inventors

• Al-Maadeed Somaya S A Ali
• Kunhoth Jayakanth
• Subramanian Nandhini
• Chaari Mohammed Zied

Classifications

• G06T7/73

  using feature-based methods · CPC title

• G01C21/005

  with correlation of navigation data from several sources, e.g. map or contour matching (G01C21/30 takes precedence) · CPC title

• H04N7/183

  for receiving images from a single remote source · CPC title

• G10L15/22

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G06V20/63

  Scene text, e.g. street names · CPC title