Device, system, and method for pictorial language for autonomous vehicle

Patent metadata

Field	Value
Publication number	US-10011285-B2
Application number	US-201615161608-A
Country	US
Kind code	B2
Filing date	May 23, 2016
Priority date	May 23, 2016
Publication date	Jul 3, 2018
Grant date	Jul 3, 2018

Abstract

The autonomous vehicle visual language system displays a plan sentence to an operator of an autonomous vehicle. The plan sentence includes visual syntax, the visual syntax being images displayed in predetermined configurations to convey specific information associated with any maneuvers the autonomous vehicle plans to execute. The visual syntax allows for a structured layering of information such that the plan sentence can display more complex information. The operator of the autonomous vehicle can process individual warnings and information more quickly, even when the operator has never seen a particular warning or element of information before. Additionally, the system 100 allows for personalization of a driver model to more closely reflect the operator's driving style, which is then implemented in the visual language displayed as the plan sentence.

First claim

The invention claimed is:

1. An autonomous vehicle pictorial language system comprising:
a plurality of sensors;
a driver preferences database;
a human-machine interface; and
a processing circuitry configured to
receive output from the plurality of sensors,
receive a driver model from the driver preferences database, the driver model predicting driver action in a plurality of driving situations based on a safety margin defined in the driver preferences, and
display a pictorial language via the human-machine interface in response to the output from the plurality of sensors and the driver model,
wherein the pictorial language includes one or more of a noun, verb, first adverb, object classifier, and second adverb to construct a plan sentence.

2. The autonomous vehicle pictorial language system of claim 1 , wherein the processing circuitry is further configured to
display the plan sentence,
receive a driver response sentence,
determine if the driver sentence received does not match the plan sentence more than a predetermined number of times,
change the first adverb of the plan sentence in response to the driver sentence not matching the plan sentence more than the predetermined number of times,
change the noun of the plan sentence in response to the driver sentence not matching the plan sentence more than the predetermined number of times, and
update the driver model in response to one or more changes to the first adverb and the noun.

3. The autonomous vehicle pictorial language system of claim 2 , wherein driver confirmation of the plan sentence causes the vehicle to execute a maneuver displayed by the plan sentence when the driver would otherwise execute the maneuver manually.

4. The autonomous vehicle pictorial language system of claim 2 , wherein the driver response sentence is constructed via output received from manual operations executed by the driver.

5. The autonomous vehicle pictorial language system of claim 2 , wherein the pictorial language is displayed as the plan sentence.

6. The autonomous vehicle pictorial language system of claim 5 , wherein a determination of the noun via the plurality of sensors and the driver model causes the human-machine interface to display a color associated with the noun based on the determination of the noun to construct the plan sentence.

7. The autonomous vehicle pictorial language system of claim 5 , wherein a determination of the first adverb via the driver model, the verb, laws associated with a location of the vehicle, and vehicle rules causes the human-machine interface to display an image associated with the first adverb based on the determination of the first adverb to construct the plan sentence.

8. The autonomous vehicle pictorial language system of claim 5 , wherein a determination of the object classifier via the plurality of sensors causes the human-machine interface to display an image associated with the object classifier based on the determination of the object classifier to construct the plan sentence.

9. The autonomous vehicle pictorial language system of claim 5 , wherein a determination of the second adverb via the plurality of sensors and the object classifier causes the human-machine interface to display an image associated with the second adverb based on the determination to construct the plan sentence.

10. The autonomous vehicle pictorial language system of claim 5 , wherein the pictorial language includes visual syntax such that visual elements share a common basis that allow for structured layering of information to display complex information.

11. The autonomous vehicle pictorial language system of claim 6 , wherein the plurality of sensors further determines road structure and sensor quality, such that the sensor quality further determines system health, to determine the noun.

12. The autonomous vehicle pictorial language system of claim 1 , wherein the plurality of sensors includes LIDAR, Radar, V2X, at least one camera, ultrasonics, and GPS.

13. The autonomous vehicle pictorial language system of claim 12 , wherein a determination of the verb via the at least one camera, the Radar, and the GPS causes the human-machine interface to display an image associated with the verb based on the determination of the verb to construct the plan sentence.

14. The autonomous vehicle pictorial language system of claim 13 , wherein the at least one camera, Radar, and GPS determine road structure, the GPS also being utilized to determine laws associated with a location of the vehicle, to determine the verb.

15. The autonomous vehicle pictorial language system of claim 14 , wherein the at least one camera is used independently or in combination with the GPS to determine laws associated with the location of the vehicle.

16. A method for displaying a pictorial language executed by a processing circuitry and comprising:
receiving output from a plurality of sensors;
receiving a driver model from a driver preferences database, the driver model predicting driver action in a plurality of driving situations based on a safety margin defined in the driver preferences; and
displaying a pictorial language via a human-machine interface in response to the output from the plurality of sensors and the driver model,
wherein the pictorial language includes one or more of a noun, verb, first adverb, object classifier, and second adverb to construct a plan sentence.

17. The method of claim 16 , further comprising:
displaying the plan sentence;
receiving a driver response sentence;
determining, via the processing circuitry, if the driver sentence received does not match the plan sentence more than a predetermined number of times;
changing the first adverb of the plan sentence in response to the driver sentence not matching the plan sentence more than the predetermined number of times;
changing the noun of the plan sentence in response to the driver sentence not matching the plan sentence more than the predetermined number of times; and
updating the driver model in response to one or more changes to the first adverb and the noun.

18. A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a computer, cause the computer to perform a method comprising:
receiving output from a plurality of sensors;
receiving a driver model from a driver preferences database, the driver model predicting driver action in a plurality of driving situations based on a safety margin defined in the driver preferences; and
displaying a pictorial language via a human-machine interface in response to the output from the plurality of sensors and the driver model,
wherein the pictorial language includes one or more of a noun, verb, first adverb, object classifier, and second adverb to construct a plan sentence.

19. The method of claim 18 , further comprising:
displaying the plan sentence;
receiving a driver response sentence;
determining if the driver sentence received does not match the plan sentence more than a predetermined number of times;
changing the first adverb of the plan sentence in response to the driver sentence not matching the plan sentence more than the predetermined number of times;
changing the noun of the plan sentence in response to the driver sentence not matching the plan sentence more than the predetermined number of times; and
updating the driver model in response to one or more changes to the first adverb and the noun.

Assignees

• Toyota Eng & Mfg North America

Inventors

• Mcnew John-Michael

Classifications

• B60W50/14 Primary

  Means for informing the driver, warning the driver or prompting a driver intervention · CPC title

• B60W2050/0088

  Adaptive recalibration · CPC title

• B60R11/0229

  for displays, e.g. cathodic tubes · CPC title

• B60W2050/146

  Display means · CPC title

• B60W2556/50

  of positioning data, e.g. GPS [Global Positioning System] data · CPC title