Systems and methods for prediction-based driver assistance

Patent metadata

Field	Value
Publication number	US-11794766-B2
Application number	US-202117501915-A
Country	US
Kind code	B2
Filing date	Oct 14, 2021
Priority date	Oct 14, 2021
Publication date	Oct 24, 2023
Grant date	Oct 24, 2023

Abstract

The present disclosure describes systems and methods for providing driver assistance. A current vehicle state of a vehicle at a current timestep and an environment map representing an environment of the vehicle at least at the current timestep are obtained. Augmented reality feedback is generated including a virtual vehicle representing a predicted future vehicle state, where the predicted future vehicle state is predicted for a given future timestep based on at least one of the current vehicle state or the current environment. The generated augmented reality feedback including the virtual vehicle is outputted to be displayed by a display device.

First claim

The invention claimed is:

1. A method comprising:
obtaining a current vehicle state of a vehicle at a current timestep and an environment map representing an environment of the vehicle at least at the current timestep;
generating augmented reality feedback including a virtual vehicle representing a predicted future vehicle state, wherein the predicted future vehicle state is predicted for a given future timestep based on at least one of the current vehicle state or the environment map, wherein generating the augmented reality feedback includes:
generating, for each one or more future timesteps, a respective predicted future vehicle state based on the current vehicle state;
determining a predicted risk at the given future timestep by applying a defined risk check to the predicted future vehicle state corresponding to the given future timestep, wherein applying the defined risk check includes:
computing a future trajectory of the vehicle from the given future timestep to a next future timestep; and
determining that the vehicle traveling over the future trajectory intersects or abuts a detected object in the environment; and

wherein the augmented reality feedback is generated responsive to determining the predicted risk; and
outputting the generated augmented reality feedback including the virtual vehicle to be displayed by a display device.

2. The method of claim 1 , wherein determining the predicted risk at the given future timestep by applying the defined risk check further comprises:
determining that the predicted future vehicle state corresponding to the given future timestep indicates a predicted vehicle location is within a defined distance threshold from the detected object in the environment.

3. The method of claim 2 , wherein the detected object is a dynamic object in the environment, wherein the predicted vehicle location is determined to be within a defined distance threshold from the dynamic object based on a predicted trajectory of the dynamic object, and wherein generating the augmented reality feedback comprises:
generating a virtual object representing the dynamic object, the virtual object being generated to be displayed in accordance with the predicted trajectory of the dynamic object.

4. The method of claim 3 , wherein the dynamic object is another vehicle in the environment, and wherein the virtual object is another virtual vehicle representing the other vehicle.

5. The method of claim 1 , wherein the detected object is a dynamic object in the environment, and wherein the vehicle traveling over the future trajectory is determined to intersect or abut the dynamic object based on a predicted trajectory of the dynamic object, and wherein generating the augmented reality feedback comprises:
generating a virtual object representing the dynamic object, the virtual object being generated to be displayed in accordance with the predicted trajectory of the dynamic object.

6. The method of claim 5 , wherein the dynamic object is another vehicle in the environment, and wherein the virtual object is another virtual vehicle representing the other vehicle.

7. The method of claim 1 , wherein determining the predicted risk at the given future timestep by applying the defined risk check further comprises at least one of:
determining that a yaw rate of the predicted future vehicle state corresponding to the given future timestep is different from a predicted trajectory heading rate, and a gradient of a yaw angle of the predicted future vehicle state corresponding to the given future timestep is different from a gradient of a steering angle of the predicted future vehicle state corresponding to the given future timestep; or
determining that a linear speed of the predicted future vehicle state corresponding to the given future timestep is greater than a defined speed threshold.

8. The method of claim 1 , wherein generating the augmented reality feedback comprises:
generating a virtual object indicating the predicted risk, the virtual object being generated to be displayed relative to the virtual vehicle.

9. The method of claim 1 , wherein generating the augmented reality feedback comprises:
further determining the predicted risk based on detecting activation of a safety system of the vehicle.

10. The method of claim 1 , wherein the augmented reality feedback is ended responsive to determining that the predicted risk has been averted.

11. The method of claim 1 , further comprising:
obtaining an updated vehicle state at a next current timestep or an updated vehicle control signal at a next current timestep;
updating the virtual vehicle representing an updated predicted future vehicle state, based on the updated vehicle state or the updated vehicle control signal; and
outputting the generated augmented reality feedback including the updated virtual vehicle to be displayed by the display device.

12. The method of claim 1 , wherein the given future timestep is selected based on a current speed of the vehicle included in the current vehicle state.

13. The method of claim 1 , wherein the display device is a head-up display of the vehicle.

14. A vehicle control system of a vehicle, the vehicle control system comprising:
a processing system; and
a memory storing instructions executable by the processing system to cause the vehicle control system to:
obtain a current vehicle state of the vehicle at a current timestep and an environment map representing an environment of the vehicle at least at the current timestep;
generate augmented reality feedback including a virtual vehicle representing a predicted future vehicle state, wherein the predicted future vehicle state is predicted for a given future timestep based on at least one of the current vehicle state or the environment map, wherein generating the augmented reality feedback includes:
generating, for each one or more future timesteps, a respective predicted future vehicle state based on the current vehicle state;
determining a predicted risk at the given future timestep by applying a defined risk check to the predicted future vehicle state corresponding to the given future timestep, wherein applying the defined risk check includes:
computing a future trajectory of the vehicle from the given future timestep to a next future timestep; and
determining that the vehicle traveling over the future trajectory intersects or abuts a detected object in the environment; and

wherein the augmented reality feedback is generated responsive to determining the predicted risk; and
output the generated augmented reality feedback including the virtual vehicle to be displayed by a display device.

15. The vehicle control system of claim 14 , wherein the instructions further cause the vehicle control system to determine the predicted risk at the given future timestep by applying the defined risk check by:
determining that the predicted future vehicle state corresponding to the given future timestep indicates a predicted vehicle location is within a defined distance threshold from the detected object in the environment.

16. The vehicle control system of claim 15 , wherein the detected object is a dynamic object in the environment, and wherein the predicted vehicle location is determined to be within a defined distance threshold from the dynamic object based on a predicted trajectory of the dynamic object, and wherein the instructions further cause the vehicle control system to generate the augmented reality feedback by:
generating a virtual object representing the dynamic object, the virtual object being generated to be displayed in acco

Assignees

• Khan Taslim Arefin
• Fan Szu Wen
• Li Wei
• Huawei Tech Co Ltd

Inventors

• Khan Taslim Arefin
• Fan Szu Wen
• Li Wei

Classifications

• B60K35/81

  for controlling displays · CPC title

• B60K35/211

  producing three-dimensional [3D] effects, e.g. stereoscopic images · CPC title

• B60K35/29

  Instruments characterised by the way in which information is handled, e.g. showing information on plural displays or prioritising information according to driving conditions · CPC title

• B60K35/233

  controlling the size or position in display areas of virtual images depending on the condition of the vehicle or the driver · CPC title

• B60K35/28

  characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver · CPC title