Method, apparatus, and computer program product for generating an overhead view of an environment from a perspective image

Patent metadata

Field	Value
Publication number	US-11720992-B2
Application number	US-202117302114-A
Country	US
Kind code	B2
Filing date	Apr 23, 2021
Priority date	Dec 21, 2018
Publication date	Aug 8, 2023
Grant date	Aug 8, 2023

Abstract

A method, apparatus and computer program product are provided for warping a perspective image into the ground plane using a homography transformation to estimate a bird's eye view in real time. Methods may include: receiving first sensor data from a first vehicle traveling along a road segment in an environment, where the first sensor data includes perspective image data of the environment, and where the first sensor data includes a location and a heading; retrieving a satellite image associated with the location and heading; applying a deep neural network to regress a bird's eye view image from the perspective image data; applying a Generative Adversarial Network (GAN) to the regressed bird's eye view image using the satellite image as a target of the GAN to obtain a stabilized bird's eye view image; and deriving values of a homography matrix between the sensor data and the established bird's eye view image.

First claim

That which is claimed:

1. An apparatus comprising at least one processor and at least one non-transitory memory including computer program code instructions, the computer program code instructions configured to, when executed, cause the apparatus to at least:
receive first sensor data from a first sensor in an environment, wherein the first sensor data comprises perspective image data of the environment, wherein the first sensor data comprises a location and a heading;
retrieve a satellite image of the environment associated with the location and heading, wherein the satellite image is captured from a position above a ground plane;
generate a bird's eye view image of the environment above the ground plane from the first sensor data by warping the first sensor data from the perspective image data of the environment based on processing of the first sensor data relative to the satellite image; and
apply a Generative Adversarial Network (GAN) to the bird's eye view image using the satellite image as a target of the GAN to obtain a stabilized bird's eye view image.

2. The apparatus of claim 1 , wherein processing of the first sensor data relative to the satellite image comprises causing the apparatus to input the first sensor data and the satellite image into a deep neural network to regress the bird's eye view image.

3. The apparatus of claim 1 , wherein the apparatus is further caused to provide for display of the stabilized bird's eye view image.

4. The apparatus of claim 1 , wherein the apparatus is further caused to:
derive values of a homography matrix between the first sensor data and the stabilized bird's eye view image; and
store the derived values of the homography matrix.

5. The apparatus of claim 4 , wherein the apparatus is further caused to:
receive second sensor data from a second sensor in a second environment, wherein the second sensor data comprises perspective image data of the second environment from a perspective of the second sensor;
apply a homography matrix including the stored, derived values to the second sensor data to warp the perspective image data of the second environment into a second ground plane;
generate a second bird's eye view of the second environment from the perspective image data of the second environment; and
provide for presentation of the second bird's eye view of the second environment on a display.

6. The apparatus of claim 5 , wherein the apparatus is further caused to augment satellite image data of the second environment with the second bird's eye view of the second environment.

7. The apparatus of claim 1 , wherein the bird's eye view image of the environment includes dynamic objects not found in a satellite image of the environment.

8. The apparatus of claim 1 , wherein causing the apparatus to retrieve a satellite image of the environment associated with the location and heading comprises causing the apparatus to identify the location in a satellite image map database and to identify a portion of the satellite image corresponding to the heading.

9. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code portions stored therein, the computer-executable program code portions comprising program code instructions configured to:
receive first sensor data from a first sensor in an environment, wherein the first sensor data comprises perspective image data of the environment, wherein the first sensor data comprises a location and a heading;
retrieve a satellite image of the environment associated with the location and heading, wherein the satellite image is captured from a position above a ground plane;
generate a bird's eye view image of the environment above the ground plane from the first sensor data by warping the first sensor data from the perspective image data of the environment based on processing of the first sensor data relative to the satellite image; and
apply a Generative Adversarial Network (GAN) to the bird's eye view image using the satellite image as a target of the GAN to obtain a stabilized bird's eye view image.

10. The computer program product of claim 9 , wherein processing of the first sensor data relative to the satellite image comprise program code instructions to input the first sensor data and the satellite image into a deep neural network to regress the bird's eye view image.

11. The computer program product of claim 9 , further comprising program code instructions to provide for display of the stabilized bird's eye view image.

12. The computer program product of claim 9 , further comprising program code instructions to:
derive values of a homography matrix between the first sensor data and the stabilized bird's eye view image; and
store the derived values of the homography matrix.

13. The computer program product of claim 12 , further comprising program code instructions to:
receive second sensor data from a second sensor in a second environment, wherein the second sensor data comprises perspective image data of the second environment from a perspective of the second sensor;
apply a homography matrix including the stored, derived values to the second sensor data to warp the perspective image data of the second environment into a second ground plane;
generate a second bird's eye view of the second environment from the perspective image data of the second environment; and
provide for presentation of the second bird's eye view of the second environment on a display.

14. The computer program product of claim 13 , further comprising program code instructions to augment satellite image data of the second environment with the second bird's eye view of the second environment.

15. The computer program product of claim 9 , wherein the bird's eye view image of the environment includes dynamic objects not found in a satellite image of the environment.

16. The computer program product of claim 9 , wherein the program code instructions to retrieve a satellite image of the environment associated with the location and heading comprise program code instructions to identify the location in a satellite image map database and to identify a portion of the satellite image corresponding to the heading.

17. A method comprising:
receiving first sensor data from a first sensor in an environment, wherein the first sensor data comprises perspective image data of the environment, wherein the first sensor data comprises a location and a heading;
retrieving a satellite image of the environment associated with the location and heading, wherein the satellite image is captured from a position above a ground plane;
generating a bird's eye view image of the environment above the ground plane from the first sensor data by warping the first sensor data from the perspective image data of the environment based on processing of the first sensor data relative to the satellite image; and
applying a Generative Adversarial Network (GAN) to the bird's eye view image using the satellite image as a target of the GAN to obtain a stabilized bird's eye view image.

18. The method of claim 17 , wherein processing of the first sensor data relative to the satellite image comprises inputting the first sensor data and the satellite image into a deep neural network to regress the bird's eye view image.

19. The method of claim 17 , further comprising providing for display of the stabilized bird's eye view.

20. The method of claim 17 , further comprising:
deriving values of a homography matrix betwee

Assignees

• Here Global Bv

Inventors

• Viswanathan Anirudh

Classifications

• G06T3/18

  Image warping, e.g. rearranging pixels individually · CPC title

• G06T1/20 Primary

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• G01C21/3667

  Display of a road map (G01C21/3614 takes precedence; guidance using 3D or perspective road maps G01C21/3635) · CPC title

• G01C21/3841

  Data obtained from two or more sources, e.g. probe vehicles · CPC title

• G01C21/3852

  Data derived from aerial or satellite images · CPC title