Generating training data from overhead view images

Patent metadata

Field	Value
Publication number	US-11288522-B2
Application number	US-201916731972-A
Country	US
Kind code	B2
Filing date	Dec 31, 2019
Priority date	Dec 31, 2019
Publication date	Mar 29, 2022
Grant date	Mar 29, 2022

Abstract

The present invention relates to a method of generating an overhead view image of an area. More particularly, the present invention relates to a method of generating a contextual multi-image based overhead view image of an area using ground map data and field of view image data.
Various embodiments of the present technology can include methods, systems and non-transitory computer readable media and computer programs configured to determine a ground map of the geographical area; receiving a plurality of images of the geographical area; process the plurality of images to select a subset of images to generate the overhead view of the geographical area; divide the ground map into a plurality of sampling points of the geographical area; and determine a color of a plurality of patches of the overhead view image from the subset of images, each patch representing each sampling point of the geographical area.

First claim

We claim:

1. A method comprising:
receiving one or more generated overhead view images of a geographical area, wherein the one or more generated overhead view images are generated by determining a ground map of the geographical area, dividing the ground map into sampling points of the geographical area, and determining a color for respective ones of the sampling points by determining an average color from correlating areas of a first image captured at a ground level over a first trajectory associated with a first vehicle and a second image captured at the ground level over a second trajectory associated with a second vehicle; and
training a computer-based model based on the one or more generated overhead view images to identify one or more map features of the geographical area.

2. The method as recited in claim 1 , further comprising extracting the one or more map features, wherein the one or more map features comprises one or more semantic features of the geographical area.

3. The method as recited in claim 1 , further comprising generating a dataset using the one or more map features, wherein the dataset comprises one or more semantic features of the geographical area.

4. The method as recited in claim 1 , further comprising receiving one or more training datasets, wherein respective ones of the one or more training datasets relate to one or more map features.

5. The method as recited in claim 1 , wherein the map features comprise at least one of lane geometries, drivable surface, lane boundaries, road markings, arrows, dashed lines, solid lines, text, crosswalks, sidewalks, traffic lights, and street furniture.

6. The method as recited in claim 4 , further comprising evaluating the first image and the second image and/or each of the one or more training datasets, wherein the first image and the second image and/or each of the one or more training datasets are evaluated against a predetermined quality threshold.

7. The method as recited in claim 6 , wherein training the computer-based model uses the first image and the second image and/or the training datasets that are determined to be below the predetermined quality threshold.

8. The method as recited in claim 6 , wherein training the computer-based model uses the first image and the second image and/or the training datasets that are determined to be above the predetermined quality threshold.

9. The method as recited in claim 1 , wherein the computer-based model comprises at least one of: machine learning models, classifiers, neural networks, U-net models and Generative Adversarial Networks.

10. The method as recited in claim 1 , further comprises: receiving the ground map of the geographical area, wherein dividing the ground map includes dividing the ground map into the sampling points using at least one of square grids, tiles, and hierarchical spatial data structures, wherein the sampling points are used as input data for training the computer-based model.

11. The method as recited in claim 1 , wherein the first image and the second image are associated with pose data and a timestamp, and wherein the first image and the second image are used as input data for training the computer-based model.

12. The method as recited in claim 1 , wherein the first image and the second image comprise limited field of view images and/or one or more sequences of images wherein the first image and the second image are used as input data for training the computer-based model.

13. The method as recited in claim 10 , wherein the ground map comprises surface and/or physical dimensions that are indicative of a drivable surface of the geographical area, and/or comprises an indication of elevation variances of a ground surface of the geographical area, and wherein the ground map is used as input data for training and/or evaluating the computer-based model.

14. The method as recited in claim 1 , wherein determining the color for the respective ones of the sampling points includes selectively separating associated pixels values from the first image and the second image using a segmentation mask that identifies and isolates a drivable road surface.

15. The method as recited in claim 10 , further comprising determining which of the first image and the second image comprises the sampling points of the geographical area.

16. The method as recited in claim 10 , wherein determining the color for the sampling points comprises filtering the first image and the second image based on a predetermined distance to the geographical area of respective ones of the sampling points.

17. The method as recited in claim 1 , wherein
determining the color for the sampling points includes determining a correlation between the sampling points of the geographical area and a color of the sampling points captured in at least one of the images.

18. A non-transitory computer-readable medium comprising computer-executable instructions which, when executed, perform a method as follows:
receiving one or more generated overhead view images of a geographical area, wherein the one or more generated overhead view images are generated by determining a ground map of the geographical area, dividing the ground map into sampling points of the geographical area, and determining a color for respective ones of the sampling points by determining an average color from correlating areas of a first image captured at a ground level over a first trajectory associated with a first vehicle and a second image captured at a ground level over a second trajectory associated with a second vehicle; and
training a computer-based model based on the one or more generated overhead view images to identify one or more map features of the geographical area.

19. A system comprising:
at least one processor and a memory storing instructions that, when executed by the at least one processor, cause the system to perform:
receiving one or more generated overhead view images of a geographical area, wherein the one or more generated overhead view images are generated by determining a ground map of the geographical area, dividing the ground map into sampling points of the geographical area, and determining a color for respective ones of the sampling points by determining an average color from correlating areas of a first image captured at a ground level over a first trajectory associated with a first vehicle and a second image captured at a ground level over a second trajectory associated with a second vehicle; and
training a computer-based model based on the one or more generated overhead view images to identify one or more map features of the geographical area.

Assignees

• Woven Planet North America Inc

Inventors

• Marschner Clemens
• Richert Wilhelm
• Schiwietz Thomas
• Zikic Darko

Classifications

• G06V20/182 Primary

  Network patterns, e.g. roads or rivers · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/803

  of input or preprocessed data · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/776

  Validation; Performance evaluation · CPC title