Personalized augmented reality vehicular assistance for color blindness condition

Patent metadata

Field	Value
Publication number	US-10423844-B2
Application number	US-201715716610-A
Country	US
Kind code	B2
Filing date	Sep 27, 2017
Priority date	Sep 27, 2017
Publication date	Sep 24, 2019
Grant date	Sep 24, 2019

Abstract

The disclosure includes embodiments for providing augmented reality (“AR”) vehicular assistance for drivers who are color blind. A method according to some embodiments includes identifying an illuminated light in a driving environment of the vehicle based on sensor data recorded by a sensor. The method includes determining a color of the illuminated light based on the sensor data. The method includes determining if the color is of a specific type. The method includes determining a vehicular action to be taken responsive to the color being of the specific type. The method includes displaying an AR overlay using the AR viewing device that visually depicts a word which describes the vehicular action to be taken.

First claim

What is claimed is:

1. A method for a vehicle including an augmented reality (“AR”) viewing device, the method comprising:
identifying an illuminated light in a driving environment of the vehicle based on sensor data recorded by a sensor;
determining a relative location of the illuminated light based on external identifier data that describes (1) an image of the illuminated light; (2) a geographic location of a connected roadway infrastructure device; and (3) a range separating the connected roadway infrastructure device from the illuminated light;
determining, based on the relative location of the illuminated light, that the illuminated light is an object to highlight using an AR overlay;
determining a color of the illuminated light based on the sensor data;
determining that the color is a specific type;
responsive to the color being the specific type, determining a vehicular action to be taken based on the color; and
displaying the AR overlay using the AR viewing device that visually depicts a word which describes the vehicular action to be taken.

2. The method of claim 1 , wherein the illuminated light is selected from a group that includes the following: a red light of a traffic light; a red light of a brake light; a yellow light of a traffic light; and a green light of a traffic light.

3. The method of claim 1 , wherein the word is selected from a group that includes the following: “STOP” for a red light of a traffic light; “BRAKE” for a red light of a brake light; “DECELERATE” for a yellow light of a traffic light; and “GO” for a green light of a traffic light.

4. The method of claim 1 , wherein the illuminated light is identified based on digital data included in a Dedicated Short Range Communication message received by the vehicle, wherein the digital data describes the illuminated light.

5. The method of claim 1 , wherein the AR viewing device is a three-dimensional heads-up display unit which is operable to display the AR overlay so that it visually appears to have three dimensions in a real-world.

6. The method of claim 1 , wherein the AR viewing device is an AR goggle.

7. The method of claim 1 , wherein the AR overlay consists of text for one word which describes the vehicular action.

8. The method of claim 1 , wherein the method is executed in real time relative to the illuminated light being identified.

9. A system of a vehicle comprising:
an augmented reality (“AR”) viewing device; and
an onboard vehicle computer system that is communicatively coupled to the AR viewing device, the onboard vehicle computer system including a non-transitory memory storing computer code which, when executed by the onboard vehicle computer system causes the onboard vehicle computer system to:
identify an illuminated light in a driving environment of the vehicle based on sensor data recorded by a sensor;
determine a relative location of the illuminated light based on external identifier data that describes (1) an image of the illuminated light; (2) a geographic location of a connected roadway infrastructure device; and (3) a range separating the connected roadway infrastructure device from the illuminated light;
determine, based on the relative location of the illuminated light, that the illuminated light is an object to highlight using an AR overlay;
determine a color of the illuminated light based on the sensor data;
determine that the color is a specific type;
responsive to the color being the specific type, determine a vehicular action to be taken based on the color; and
display the AR overlay using the AR viewing device that visually depicts a word which describes the vehicular action to be taken.

10. The system of claim 9 , wherein the illuminated light is identified based on measurements of the driving environment recorded by one or more onboard sensors of the vehicle.

11. The system of claim 9 , wherein the illuminated light is identified based on digital data included in a Dedicated Short Range Communication message received by the vehicle, wherein the digital data describes the illuminated light.

12. The system of claim 9 , wherein the AR viewing device is a three-dimensional heads-up display unit which is operable to display the AR overlay so that it visually appears to have three dimensions in a real-world.

13. The system of claim 9 , wherein the AR viewing device is an AR goggle.

14. The system of claim 9 , wherein the AR overlay consists of text for one word which describes the vehicular action.

15. The system of claim 9 , wherein the AR overlay is displayed in real time relative to the illuminated light being identified.

16. A computer program product comprising a non-transitory memory of an onboard vehicle computer system of a vehicle storing computer-executable code that, when executed by the onboard vehicle computer system, causes the onboard vehicle computer system to:
identify an illuminated light in a driving environment of the vehicle based on sensor data recorded by a sensor;
determine a relative location of the illuminated light based on external identifier data that describes (1) an image of the illuminated light; (2) a geographic location of a connected roadway infrastructure device; and (3) a range separating the connected roadway infrastructure device from the illuminated light;
determine, based on the relative location of the illuminated light, that the illuminated light is an object to highlight using an AR overlay;
determine a color of the illuminated light based on the sensor data;
determine that the color is a specific type;
responsive to the color being the specific type, determine a vehicular action to be taken based on the color; and
display the AR overlay using the AR viewing device that visually depicts a word which describes the vehicular action to be taken.

17. The computer program product of claim 16 , wherein the illuminated light is identified based on measurements of the driving environment recorded by one or more onboard sensors of the vehicle.

18. The computer program product of claim 16 , wherein the illuminated light is identified based on digital data included in a Dedicated Short Range Communication message received by the vehicle, wherein the digital data describes the illuminated light.

19. The computer program product of claim 16 , wherein the AR viewing device is a three-dimensional heads-up display unit which is operable to display the AR overlay so that it visually appears to have three dimensions in a real-world.

20. The computer program product of claim 16 , wherein the AR overlay consists of text for one word which describes the vehicular action.

Assignees

• Toyota Motor Co Ltd

Inventors

• Dai Siyuan
• Arechiga Nikos
• Lin Chung-Wei
• Shiraishi Shinichi

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G08G1/09623

  Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle (G08G1/0967 takes precedence) · CPC title

• G02B2027/0141

  characterised by the informative content of the display · CPC title

• G08G1/095

  Traffic lights · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title