Method and apparatus for displaying virtual route

Patent metadata

Field	Value
Publication number	US-11204253-B2
Application number	US-202016910942-A
Country	US
Kind code	B2
Filing date	Jun 24, 2020
Priority date	Nov 9, 2017
Publication date	Dec 21, 2021
Grant date	Dec 21, 2021

Abstract

A method and apparatus for displaying a virtual route estimates a position of a vehicle based on sensing data received asynchronously from sensors, and outputs a three-dimensional (3D) virtual route generated by registering a driving environment model corresponding to a driving environment of the vehicle and the position of the vehicle in map information.

First claim

What is claimed is:

1. A method of displaying a virtual route, the method comprising:
estimating first and second positions of a vehicle based on sensing data received from a first sensor and a second sensor;
generating a three-dimensional (3D) virtual route by registering, in map information, a driving environment model corresponding to a driving environment of the vehicle and the first and second positions of the vehicle; and
outputting the 3D virtual route,
wherein a sensing speed of the first sensor and a sensing speed of the second sensor are not synchronized.

2. The method of claim 1 , wherein the estimating comprises estimating the first and second positions without synchronizing the sensing speeds of the first sensor and the second sensor.

3. The method of claim 1 , wherein a sensing speed of the first sensor is less than a sensing speed of the second sensor.

4. The method of claim 1 , wherein the estimating comprises:
estimating the first position based on first sensing data received from the first sensor; and
estimating the second position of the vehicle based on second sensing data received from the second sensor.

5. The method of claim 4 , further comprising updating the first position of the vehicle to the second position in response to the estimating of the second position.

6. The method of claim 1 , wherein the first and second sensors comprise any two of a global positioning system (GPS) sensor, an inertial measurement unit (IMU) sensor, an on-board diagnostics (OBD) sensor, and a camera sensor.

7. The method of claim 1 , wherein the sensing data are sampled separately for each of the first and second sensors.

8. The method of claim 1 , wherein the estimating comprises:
inserting, into each piece of the sensing data, information related to a processing time at which the corresponding piece of the sensing data is processed; and
estimating the first and second positions of the vehicle based on the sensing data, each including the information related to the processing time.

9. The method of claim 1 , wherein the estimating comprises:
estimating the first position of the vehicle based on first sensing data received at a first time from the first sensor;
updating the first position of the vehicle to the second position based on second sensing data received at a second time from the first sensor or the second sensor; and
updating the second position of the vehicle to a third position based on third sensing data received at a third time from the first sensor, or the second sensor, or a third sensor.

10. The method of claim 1 , further comprising:
measuring an odometry based on image data acquired from a camera sensor among the first and second sensors,
wherein the estimating comprises estimating either one or both of the first and second positions of the vehicle based on the odometry.

11. The method of claim 10 , wherein the estimating based on the sensing data and the odometry comprises:
estimating the either one or both of the first and second positions of the vehicle based on either one piece of the sensing data, except for the image data; and
correcting the either one or both of the first and second positions of the vehicle based on the odometry.

12. The method of claim 1 , wherein the generating of the 3D virtual route comprises:
generating a segmentation image based on image data acquired from a camera sensor among the sensors;
detecting objects included in the segmentation image;
generating the driving environment model based on depth values of the objects and a driving lane of the vehicle identified from the objects; and
generating the 3D virtual route by registering, in the map information, the driving environment model and the first and second positions of the vehicle.

13. The method of claim 1 , further comprising:
transforming the 3D virtual route to match to a viewpoint of a driver of the vehicle,
wherein the outputting comprises displaying the transformed 3D virtual route.

14. The method of claim 13 , wherein the outputting of the transformed 3D virtual route comprises displaying the transformed 3D virtual route through a head-up display (HUD) of the vehicle.

15. The method of claim 13 , wherein the transforming comprises:
tracking the viewpoint of the driver by tracking 3D positions of both eyes of the driver;
predicting a transformation relation between the viewpoint of the driver and a virtual image displayed through the HUD based on the 3D positions of both the eyes of the user; and
transforming the 3D virtual route based on the transformation relation.

16. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform the method of claim 1 .

17. An apparatus for displaying a virtual route, the apparatus comprising:
a first sensor and a second sensor configured to sense sensing data;
one or more processors configured to
estimate first and second positions of a vehicle based on the sensing data, and
generate a three-dimensional (3D) virtual route by registering, in map information,

a driving environment model corresponding to a driving environment of the vehicle and
the first and second positions of the vehicle; and
a display configured to display the 3D virtual route,
wherein a sensing speed of the first sensor and a sensing speed of the second sensor are not synchronized.

18. The apparatus of claim 17 , wherein the one or more processors are further configured to estimate the first and second positions without synchronizing the sensing speeds of the first sensor and the second sensor.

19. The apparatus of claim 17 , wherein a sensing speed of the first sensor is less than a sensing speed of the second sensor.

20. The apparatus of claim 17 , wherein the one or more processors are further configured to estimate the first position based on first sensing data received from the first sensor; and estimate the second position of the vehicle based on second sensing data received from the second sensor.

21. The apparatus of claim 20 , further the one or more processors are further configured to update the first position of the vehicle to the second position in response to the estimating of the second position.

22. The apparatus of claim 17 , wherein the first and second sensors comprise any two of:
a global positioning system (GPS) sensor configured to measure an absolute route of the vehicle;
an inertial measurement unit (IMU) sensor configured to measure a relative route of the vehicle;
an on-board diagnostics (OBD) sensor configured to measure a driving distance of the vehicle; and
a camera sensor configured to capture image data including the driving environment of the vehicle.

23. The apparatus of claim 22 , wherein the sensing data are sampled separately for each of the first and second sensors.

24. The apparatus of claim 17 , wherein the one or more processors are further configured to insert, into each piece of the sensing data, information related to a processing time at which the corresponding piece of the sensing data is processed, and to estimate the first and second positions of the vehicle based on the sensing data, each including the information related to the processing time.

25. The apparatus of claim 17 , wherein the one or more processors are further configured to estimate the first position of the vehicle based on first

Assignees

• Samsung Electronics Co Ltd

Inventors

• Jung Kyungboo
• Kang Nahyup
• Lee Keechang
• Hong Sunghoon

Classifications

• G01C21/3819

  Road shape data, e.g. outline of a route · CPC title

• G01C21/3867

  Geometry of map features, e.g. shape points, polygons or for simplified maps · CPC title

• G01C21/3848

  Data obtained from both position sensors and additional sensors · CPC title

• G01C21/188

  for accumulated errors, e.g. by coupling inertial systems with absolute positioning systems · CPC title

• G01C21/3635 Primary

  Guidance using 3D or perspective road maps · CPC title