Double-blinded, randomized trial of augmented reality low-vision mobility and grasp aid

Patent metadata

Field	Value
Publication number	US-12440416-B2
Application number	US-201917298589-A
Country	US
Kind code	B2
Filing date	Dec 2, 2019
Priority date	Nov 30, 2018
Publication date	Oct 14, 2025
Grant date	Oct 14, 2025

Abstract

An augmented reality system for providing depth perspective includes a sensor system that provides spatial data of objects in a surrounding environment of a user. A computer processor system calculates spatial information of the objects from the spatial data received the sensor system. The computer processor system determines a depth-to-color mapping in which distance of objects from the user is mapped to a predetermined viewable representation. The system also includes a head mountable display that displays the depth-to-color mapping to the user. Characteristically, distances of the objects from the user are rendered to allow at least partial viewability of the object by the user. A method utilizing the augmented reality system is also provided.

First claim

What is claimed is:

1. An augmented reality system for providing depth perspective to a low-vision user, the augmented reality system comprising:
a sensor system that provides spatial data of objects in a surrounding environment of the low-vision user, wherein the sensor system includes at least one electromagnetic sensor, optical sensor, or video sensor;
a computer processor system that calculates spatial information of the objects from the spatial data received from the sensor system, the computer processor system determining a depth-to-color mapping in which distance of objects from the low-vision user is mapped to a predetermined viewable representation, wherein the depth-to-color mapping includes a colored wireframe with edge-enhancement; and
a head-mountable display that displays the depth-to-color mapping to a low-vision user, wherein distances of the objects from the low-vision user are rendered to allow at least partial viewability of the objects by the low-vision user and wherein the depth-to-color mapping assists in identifying objects by applying a pseudocolor map thereby facilitating navigation and grasp by the low-vision user, the pseudocolor map including discrete color changes to indicate varying distances of objects thereby ensuring partial viewability and object detection for the low-vision user, the computer processor system further being configured to construct a triangular point mesh using a geometric shader rather than continuously rendering a surface over the real world, wherein only an object's edges are represented with a wireframe and therefore do not obstruct text written on an object with a color overlay, and wherein the depth-to-color mapping is limited to objects within a maximum distance of 6 feet from the low-vision user to prevent sensory overload.

2. The augmented reality system of claim 1 where the depth-to-color mapping assists in identifying objects.

3. The augmented reality system of claim 1 wherein the sensor system and the head-mountable display are integrated into a single device.

4. The augmented reality system of claim 1 wherein the sensor system, the computer processor system and the head-mountable display are integrated into a single device.

5. The augmented reality system of claim 1 wherein the sensor system and the head-mountable display are integrated into a first device and the computer processor system being a separate device in communication with the first device.

6. The augmented reality system of claim 1 wherein the predetermined viewable representation further includes a component selected from the group consisting of a frequency map in which there are different frequencies for different distances, a brightness map in which there are different brightness for different distances, size map in which closer objects are bigger, and combinations thereof.

7. The augmented reality system of claim 1 wherein the predetermined viewable representation is a predetermined color.

8. The augmented reality system of claim 7 wherein the predetermined color is a pseudocolor or false-color with a plurality of discrete color changes based on distance from the low-vision user.

9. The augmented reality system of claim 8 wherein the plurality of discrete color changes includes from 3 to 10 color changes.

10. The augmented reality system of claim 1 wherein the sensor system includes a plurality of electromagnetic, optical, or video sensors.

11. The augmented reality system of claim 10 wherein spatial data from the plurality of electromagnetic, optical, or video sensors is analyzed by sensor fusion to provide the spatial information.

12. The augmented reality system of claim 1 wherein the sensor system includes a depth camera.

13. The augmented reality system of claim 12 wherein the sensor system includes at least one greyscale environment sensing camera that works with the depth camera to track the surrounding environment.

14. The augmented reality system of claim 1 wherein the sensor system includes a video camera and a light sensor.

15. The augmented reality system of claim 1 wherein the computer processor system includes a CPU and a GPU.

16. A method for providing depth perspective to a low-vision user, the method comprising:
receiving spatial data for objects in a surrounding environment of the low-vision user from at least one of an electromagnetic sensor, optical sensor, or video sensor;
calculating spatial information of the objects from spatial data, the spatial information including a depth-to-color mapping in which distance of objects from the low-vision user is mapped to a predetermined viewable representation, wherein the depth-to-color mapping includes a colored wireframe having a color encoded distance;
constructing a triangular point mesh using a geometric shader rather than continuously rendering a surface over the real world, wherein only an object's edges are represented with a wireframe and therefore do not obstruct text written on an object with a color overlay; and

displaying the depth-to-color mapping to the low-vision user on a head-mountable display wherein distances of the objects from the low-vision user are rendered to allow at least partial viewability of the objects by the low-vision user and wherein the depth-to-color mapping assists in identifying objects by applying a pseudocolor map thereby facilitating navigation and grasp by the low-vision user, the pseudocolor map including discrete color changes to indicate varying distances of objects thereby ensuring partial viewability and object detection for the low-vision user; and
limiting the depth-to-color mapping to objects within a maximum distance of 6 feet from the low-vision user to prevent sensory overload.

17. The method of claim 16 wherein the predetermined viewable representation is selected from the group consisting of pseudocolor map, a frequency map in which there are different frequencies for different distances, a brightness map in which there are different brightness for different distances, size map in which closer objects are bigger, and combinations thereof.

18. The method of claim 16 wherein the depth-to-color mapping assists in object identification.

19. The method of claim 16 wherein the predetermined viewable representation is a predetermined color.

20. The method of claim 19 wherein the predetermined color is a pseudocolor or false-color with a plurality of discrete color changes based on distance from the low-vision user.

21. The method of claim 20 wherein the plurality of discrete color changes includes from 3 to 10 changes.

22. The augmented reality system of claim 15 wherein the computer processor system further includes a custom chipset or an application-specific integrated chip.

Assignees

• Univ Southern California

Inventors

• Humayun Mark S
• Angelopoulos Anastasios

Classifications

• G06T2219/2012

  Colour editing, changing, or manipulating; Use of colour codes · CPC title

• G06T2210/41

  Medical · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G02B2027/0127

  comprising devices increasing the depth of field · CPC title