Method, device and computer-readable storage medium with instructions for controlling a display of an augmented-reality head-up display device for a transportation vehicle

Patent metadata

Field	Value
Publication number	US-11325470-B2
Application number	US-201816649468-A
Country	US
Kind code	B2
Filing date	Aug 28, 2018
Priority date	Sep 21, 2017
Publication date	May 10, 2022
Grant date	May 10, 2022

Abstract

A method, device, and computer-readable storage medium with instructions for controlling a display of an augmented-reality head-up display device for a transportation vehicle. A position of the transportation vehicle is determined relative to a map; at least one reference point is extracted from the map, the at least one reference point is obtained from an area within a visual range of a driver of the transportation vehicle; transforming the at least one reference point into transportation vehicle coordinates; generating a virtual object for display by the augmented-reality head-up display device, considering the at least one transformed reference point; and outputting the virtual object for display by the augmented-reality head-up display device.

First claim

The invention claimed is:

1. A method for controlling a display by an augmented-reality head-up display apparatus for a transportation vehicle, the method comprising:
determining a pose of the transportation vehicle relative to a map to ascertain a position and orientation of the transportation vehicle, wherein the pose determination correlates one or more features that have been detected by a sensor system of the transportation vehicle with one or more features of the map, wherein the map includes previously measured road plane data, wherein the map contains information concerning topography, wherein an exact position and orientation of the sensor system of the transportation vehicle is subject to manufacturing tolerances and differs between different transportation vehicles, wherein the sensor system is calibrated during production, or its position and orientation are estimated in the course of operation whereby resulting parameters are stored in the transportation vehicle;
extracting at least one reference point from the map by taking into consideration the determined pose of the transportation vehicle relative to the map, wherein the at least one reference point originates from a ground surface road plane in a visual range of a driver of the transportation vehicle;
transforming the at least one ground surface road plane reference point into transportation vehicle coordinates based on a transportation vehicle reference coordinate system or target coordinate system that is specific to the transportation vehicle, thereby removing inaccuracies relating to the sensor system of the transportation vehicle;
generating a virtual object for display by the augmented-reality head-up display apparatus based at least in part on analysis of the at least one ground surface road plane transformed reference point; and
outputting the virtual object for display by the augmented-reality head-up display apparatus, wherein depiction of the virtual object is positioned so as to be placed directly onto the road plane in the augmented-reality head-up display apparatus,
wherein the at least one ground surface road plane reference point is extracted using a previously calculated geometric abstraction of a topography that is stored in the map, and
wherein the geometric abstraction of the topography comprises one-dimensional or two-dimensional polynomials or a grid.

2. The method of claim 1 , wherein the at least one ground surface road plane reference point is extracted from a map region of the map in the direction of travel of the transportation vehicle.

3. The method of claim 2 , wherein the map region comprises a ground region and an extended region based on a velocity of the transportation vehicle.

4. The method of claim 1 , wherein the generating of the virtual object includes transforming graphical elements of the virtual object onto the ground surface and distorting the graphical elements of the virtual object so as to appear to be placed directly onto the road plane in the augmented-reality head-up display apparatus.

5. A non-transitory computer-readable storage medium having instructions that, when executed by a computer, control the computer to perform a method for controlling a display by an augmented-reality head-up display apparatus for a transportation vehicle, the method comprising:
determining a pose of the transportation vehicle relative to a map to ascertain a position and orientation of the transportation vehicle, wherein the pose determination correlates one or more features that have been detected by a sensor system of the transportation vehicle with one or more features of the map, wherein the map includes previously measured road plane data, wherein the map contains information concerning topography, wherein an exact position and orientation of the sensor system of the transportation vehicle is subject to manufacturing tolerances and differs between different transportation vehicles, wherein the sensor system is calibrated during production, or its position and orientation are estimated in the course of operation whereby resulting parameters are stored in the transportation vehicle;
extracting at least one reference point from the map by taking into consideration the determined pose of the transportation vehicle relative to the map, wherein the at least one reference point originates from a ground surface road plane in a visual range of a driver of the transportation vehicle;
transforming the at least one ground surface road plane reference point into transportation vehicle coordinates based on a transportation vehicle reference coordinate system or target coordinate system that is specific to the transportation vehicle, thereby removing inaccuracies relating to the sensor system of the transportation vehicle generating a virtual object for display by the augmented-reality head-up display apparatus based at least in part on analysis of the at least one ground surface road plane transformed reference point; and
outputting the virtual object for display by the augmented-reality head-up display apparatus, wherein depiction of the virtual object is positioned so as to be placed directly onto the road plane in the augmented-reality head-up display apparatus,
wherein the at least one ground surface road plane reference point is extracted using a previously calculated geometric abstraction of a topography that is stored in the map, and
wherein the geometric abstraction of the topography comprises one-dimensional or two-dimensional polynomials or a grid.

6. An apparatus for controlling a display by an augmented-reality head-up display apparatus for a transportation vehicle, the apparatus comprising:
a pose determination unit configured to determine a pose of the transportation vehicle relative to a map to ascertain a position and orientation of the transportation vehicle, wherein the pose determination correlates one or more features that have been detected by a sensor system of the transportation vehicle with one or more features of the map, wherein the map includes previously measured road plane data, wherein the map contains information concerning topography, wherein an exact position and orientation of the sensor system of the transportation vehicle is subject to manufacturing tolerances and differs between different transportation vehicles, wherein the sensor system is calibrated during production, or its position and orientation are estimated in the course of operation whereby resulting parameters are stored in the transportation vehicle;
an extraction unit configured to extract at least one reference point from the map by taking into consideration the determined pose of the transportation vehicle relative to the map, wherein the at least one reference point originates from a ground surface road plane in a visual range of a driver of the transportation vehicle;
a transformation unit configured to transform the at least one ground surface road plane reference point into transportation vehicle coordinates based on a transportation vehicle reference coordinate system or target coordinate system that is specific to the transportation vehicle, thereby removing inaccuracies relating to the sensor system of the transportation vehicle;
a graphics unit to generate a virtual object for display by the augmented-reality head-up display apparatus based at least in part on analysis of the at least one transformed ground surface road plane reference point; and
an output to output the virtual object for display by the augmented-reality head-up display apparatus, wherein depiction of the virtual object is positioned so as to be placed directly onto the road plane in the augmented-reality head-up display apparatus,
wherein the at least one ground surface road plane reference point is extracted using a previously calc

Assignees

• Volkswagen Ag

Inventors

• Max Stephan
• Wuttke Ulrich

Classifications

• G06T7/74 Primary

  involving reference images or patches · CPC title

• B60K35/28

  characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver · CPC title

• B60K35/23

  Head-up displays [HUD] (optical aspects of head-up displays G02B27/01) · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G01C21/365 Primary

  Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself · CPC title