Multimodal image perception system and method

Patent metadata

Field	Value
Publication number	US-10600336-B2
Application number	US-201615284505-A
Country	US
Kind code	B2
Filing date	Oct 3, 2016
Priority date	Oct 2, 2015
Publication date	Mar 24, 2020
Grant date	Mar 24, 2020

Abstract

A real-time multimodal image perception system to transform the standard lab blood smear image for persons with BVI to perceive, employing a combination of auditory, haptic, and vibrotactile feedbacks. These sensory feedbacks are used to convey visual information in appropriate perceptual channels, thus creating a palette of multimodal, sensorial information. A Bayesian network is provided to characterize images through two groups of features of interest: primary and peripheral features. A method is provided for optimal matching between primary features and sensory modalities.

First claim

The invention claimed is:

1. A method for enabling visually impaired users to interpret data, comprising:
receiving an input image using a computer processor, the image representing a biological sample;
receiving a navigation command from a visually impaired user, the navigation command comprising instructions to direct the processor to evaluate an area within the image;
automatically extracting a plurality of features from the input image to acquire at least one extracted image feature based on the navigation command using the processor;
developing a Bayesian network using the processor, the Bayesian network is configured to characterize the input image through the two groups of features of interest, the features comprising primary features and peripheral features; and
outputting an expression of the plurality of features through at least one sensory modality operatively connected to the computer processor to allow a visually-impaired user to interpret the image features via the at least one sensory modality, wherein the at least one modality comprises at least one of auditory, haptic, and vibrotactile.

2. The method of claim 1 , wherein the plurality of features comprises image location, intensity, texture, shape color, size, and opacity.

3. The method of claim 1 , wherein a linear assignment problem is utilized to assign the image features to the plurality of sensory modalities.

4. The method of claim 1 , wherein a quadtratic assignment problem is utilized to assign the image features to the plurality of sensory modalities.

5. The method of claim 1 , wherein the plurality of output devices comprise at least two hand devices to allow the user to use both hands to interface with the system.

6. The method of claim 1 , wherein the navigation command is received from a stylus or a gripper operatively connected to the computer processor.

7. The method of claim 1 , wherein a haptic device is used by a first hand of the user to navigate the image.

8. The method of claim 7 , wherein a second hand of the user is used to interact with the vibrotactile device to perceive the image features.

9. A system for enabling visually impaired users to interpret data, comprising:
an image input device for receiving an image, the image representing a magnified biological sample;
a user input device which is configured to allow the user to navigate within the image;
a plurality of output devices configured to output a plurality of sensory modalities to a visually impaired user; and
a computer processing unit operatively connected to the plurality of output devices, the computer processing unit configured to:
receive an input image;
receive a navigation command from a visually impaired user, the navigation command comprising instructions to direct the processor to evaluate an area within the image;
extract a plurality of features from the input image to acquire at least one extracted image feature based on the navigation command;
develop a Bayesian network, the Bayesian network is configured to characterize the input image through the two groups of features of interest, the features comprising primary features and peripheral features; and
output an expression of the plurality of features through at least one sensory modality to allow a visually-impaired user to interpret the image features via the at least one sensory modality, wherein the at least one modality comprises at least one of auditory, haptic, and vibrotactile.

10. The system of claim 9 , wherein the plurality of features comprises image location, intensity, texture, shape color, size, and opacity.

11. The system of claim 9 , wherein a linear assignment problem is utilized to assign the image features to the plurality of sensory modalities.

12. The system of claim 9 , wherein a quadtratic assignment problem is utilized to assign the image features to the plurality of sensory modalities.

13. The system of claim 9 , wherein the plurality of output devices comprise at least two hand devices to allow the user to use both hands to interface with the system.

14. The system of claim 9 , wherein the navigation command is received from a stylus or a gripper operatively connected to the computer processor.

15. The system of claim 9 , wherein a haptic device is used by a first hand of the user to navigate the image.

16. The system of claim 15 , wherein a second hand of the user is used to interact with the vibrotactile device to perceive the image features.

Assignees

• Purdue Research Foundation

Inventors

• Duerstock Bradley S
• Wachs Juan P
• Zhang Ting
• Williams Greg

Classifications

• G09B21/007 Primary

  using both tactile and audible presentation of the information · CPC title

• G09B21/008 Primary

  using visual presentation of the information for the partially sighted · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06K9/6278

  Physics · mapped topic

• G06K9/4604

  Physics · mapped topic