Detection of invariant features for localization

Patent metadata

Field	Value
Publication number	US-9940729-B1
Application number	US-201615355727-A
Country	US
Kind code	B1
Filing date	Nov 18, 2016
Priority date	Nov 18, 2016
Publication date	Apr 10, 2018
Grant date	Apr 10, 2018

Abstract

A first image and a second image are provided to a trained neural network. The first image comprises one or more static features and the second image comprises at least one of the one or more static features. A static feature is identified in both the first and second images by a branch of the trained neural network. A three dimensional image comprising the identified static feature is generated and three dimensional geometric information/data related to the static feature is extracted and stored in association with a tile of a digital map. A set of training images may be used to train the trained neural network comprises training image subsets comprising two or more images that substantially overlap that were (a) captured at different times; (b) captured under different (i) weather conditions, (ii) lighting conditions, or (iii) weather and lighting conditions; or both a and b.

First claim

That which is claimed:

1. A method comprising:
providing to a trained neural network a first image, wherein the first image (a) is captured by an image capture device at a first location and at a first pose, (b) is associated with geophysical coordinates corresponding to the first location, and (c) comprises one or more static features;
providing to the trained neural network a second image, wherein the second image (a) is captured by an image capture device at a second location and at a second pose, (b) is associated with geophysical coordinates corresponding to the second location, and (c) comprises at least one of the one or more static features;
identifying the at least one of the one or more static features in both the first and second images with a branch of the trained neural network;
generating a three dimensional image comprising the at least one identified static feature and based on the first and second images;
extracting a three dimensional geometry of the at least one identified static feature;
encoding the extracted three dimensional geometry as an array;
based on the first location and second location, determining a static feature location for the at least one identified static feature; and
storing the array in association with a map tile, wherein the map tile is selected based on the static feature location.

2. A method according to claim 1 , wherein a set of training images used to train the trained neural network comprises training image subsets comprising two or more images that substantially overlap that were captured at different times.

3. A method according to claim 1 , wherein a training set of images used to train the trained neural network comprises training image subsets comprising two or more images that substantially overlap that were captured under different (a) weather conditions, (b) lighting conditions, or (c) weather and lighting conditions.

4. A method according to claim 1 , wherein the three dimensional image is a binary feature map.

5. A method according to claim 1 , wherein the array comprises shape information for the corresponding static feature.

6. A method according to claim 1 , wherein at least one training image of a set of training images used to train the trained network comprises at least one pixel that is masked, the at least one masked pixel corresponding to a non-static feature in the training image.

7. A method according to claim 1 , wherein (a) the first location differs from the second location, (b) the first pose differs from the second pose, or (c) both the first location differs from the second location and the first pose differs from the second pose.

8. A method according to claim 1 , wherein the static feature of the one or more static features is a generic static feature.

9. A method according to claim 1 , wherein static feature information is stored in association with the map tile, the static feature information comprising the array and the static feature location.

10. A method according to claim 9 , wherein static feature information is stored in association with a road segment of the map tile.

11. A method according to claim 9 , wherein (a) a static feature orientation is determined from at least one of the first location, first pose, second location, or second pose and (b) the static feature information further comprises the static feature orientation.

12. An apparatus comprising at least one processor and at least one memory storing computer program code, the at least one memory and the computer program code configured to, with the processor, cause the apparatus to at least:
provide to a trained neural network a first image, wherein the first image (a) is captured by an image capture device at a first location and at a first pose, (b) is associated with geophysical coordinates corresponding to the first location, and (c) comprises one or more static features;
provide to the trained neural network a second image, wherein the second image (a) is captured by an image capture device at a second location and at a second pose, (b) is associated with geophysical coordinates corresponding to the second location, and (c) comprises at least one of the one or more static features;
identify the at least one of the one or more static features in both the first and second images with a branch of the trained neural network;
generate a three dimensional image comprising the at least one identified static feature and based on the first and second images;
extract a three dimensional geometry of the at least one identified static feature;
encode the extracted three dimensional geometry as an array;
based on the first location and second location, determine a static feature location for the at least one identified static feature; and
store the array in association with a map tile, wherein the map tile is selected based on the static feature location.

13. An apparatus according to claim 12 , wherein a set of training images used to train the trained neural network comprises training image subsets comprising two or more images that substantially overlap that were captured at different times.

14. An apparatus according to claim 12 , wherein a training set of images used to train the trained neural network comprises training image subsets comprising two or more images that substantially overlap that were captured under different (a) weather conditions, (b) lighting conditions, or (c) weather and lighting conditions.

15. An apparatus according to claim 12 , wherein at least one training image of a set of training images used to train the trained network comprises at least one pixel that is masked, the at least one masked pixel corresponding to a non-static feature in the training image.

16. An apparatus according to claim 12 , wherein (a) the first location differs from the second location, (b) the first pose differs from the second pose, or (c) both the first location differs from the second location and the first pose differs from the second pose.

17. An apparatus according to claim 12 , wherein static feature information is stored in association with the map tile, the static feature information comprising the array and the static feature location.

18. An apparatus according to claim 17 , wherein static feature information is stored in association with a road segment of the map tile.

19. An apparatus according to claim 17 , wherein (a) a static feature orientation is determined from at least one of the first location, first pose, second location, or second pose and (b) the static feature information further comprises the static feature orientation.

20. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising program code instructions configured to:
provide to a trained neural network a first image, wherein the first image (a) is captured by an image capture device at a first location and at a first pose, (b) is associated with geophysical coordinates corresponding to the first location, and (c) comprises one or more static features;
provide to the trained neural network a second image, wherein the second image (a) is captured by an image capture device at a second location and at a second pose, (b) is associated with geophysical coordinates corresponding to the second location, and (c) comprises at least one of the one or more static features;
identify the at least one of the one or more static fea

Assignees

• Here Global Bv

Inventors

• Kwant Richard
• Mittal Anish
• Viswanathan Anirudh

Classifications

• G06T7/0042

  Physics · mapped topic

• G06T7/60 Primary

  Analysis of geometric attributes · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10004

  Still image; Photographic image · CPC title

• G06V10/82 Primary

  using neural networks · CPC title