Unsupervised approach to environment mapping at night using monocular vision

Patent metadata

Field	Value
Publication number	US-10424079-B2
Application number	US-201715479617-A
Country	US
Kind code	B2
Filing date	Apr 5, 2017
Priority date	Apr 5, 2017
Publication date	Sep 24, 2019
Grant date	Sep 24, 2019

Abstract

A trained feature network receives an image captured under low illumination conditions and pose data corresponding to the image. The trained feature network identifies a feature within the image and analyzes the image to extract feature information corresponding to the feature from the image. Based on the image and the pose data, geo-location information corresponding to the feature is determined. The geo-location information is appended to the extracted feature information. The feature information is stored as part of a feature map layer of a digital map. At least a portion of the digital map is provided to a routing and navigation system, for example, for performing vehicle localization under the particular condition.

First claim

The invention claimed is:

1. A map provider system configured for providing a feature map for vehicle localization under low illumination level conditions, the map provider system comprising:
at least one memory, the at least one memory storing a digital map;
at least one communications interface configured to enable communication with one or more routing and navigation systems;
at least one processor configured to:
operate a trained deep neural network, the trained deep neural network configured to:
receive an image captured under low illumination level conditions and pose data corresponding to the image;
identify a feature within the image;
analyze the image to extract feature information corresponding to the feature from the image;
determine, based on the image and the pose data, geo-location information corresponding to the feature; and
append the geo-location information to the feature information; and

store the feature information as part of a feature map layer of the digital map; and
cause the at least one communications interface to provide at least a portion of the digital map to a routing and navigation system, wherein the routing and navigation system is configured to (a) perform a localization determination based on the feature map layer of the portion of the digital map and (b) make a routing decision based on the localization determination.

2. A map provider system according to claim 1 , wherein, to train the deep neural network, the at least one processor is configured to:
cause the deep neural network to receive a sequence of training images, the sequence of training images captured under particular conditions;
analyze, by the deep neural network, a portion of the sequence of training images, the sequence of training images comprising a particular training image and the portion of the sequence of training images comprising two or more training images of the sequence of training images that are not the particular training image;
identify, by the deep neural network, one or more features based on the analysis of the portion of sequence of training images;
generate, by the deep neural network, an encoded representation of the particular training image;
determine a loss function based on an analysis of the particular training image and the encoded representation of the particular training image; and
update a network weight of the deep neural network based at least in part on the loss function.

3. A map provider system according to claim 2 , wherein the deep neural network is formatted as an auto-encoder representation.

4. A map provider system according to claim 2 , wherein the sequence of training images is a temporal sequence and the particular training image is a middle image of the temporal sequence.

5. A map provider system according to claim 2 , wherein (a) a sequence of pose information is provided to the deep neural network, (b) each instance of the sequence of pose information corresponds to a training image of the sequence of training images, and (c) the one or more features are identified based at least in part on motion indicated by the sequence of pose information.

6. A map provider system according to claim 1 , wherein the image is captured at nighttime.

7. A method comprising:
receiving, by a trained deep neural network, an image captured under low illumination level conditions and pose data corresponding to the image;
identifying, by the trained deep neural network, a feature within the image;
analyzing the image with the trained deep neural network to extract feature information corresponding to the feature from the image;
determining, by the trained deep neural network and based on the image and the pose data, geo-location information corresponding to the feature;
appending, by the deep neural network, the geo-location information to the feature information;
storing the feature information as part of a feature map layer of a digital map; and
providing at least a portion of the digital map to a routing and navigation system.

8. A method according to claim 7 , wherein training the deep neural network comprises:
receiving, by the deep neural network, a sequence of training images, the sequence of training images captured under particular conditions;
analyzing, by the deep neural network, a portion of the sequence of training images, the sequence of training images comprising a particular training image and the portion of the sequence of training images comprising two or more training images of the sequence of training images that are not the particular training image;
identifying, by the deep neural network, one or more features based on the analysis of the portion of sequence of training images;
generating, by the deep neural network, an encoded representation of the particular training image;
determining a loss function based on an analysis of the particular training image and the encoded representation of the particular training image; and
updating a network weight of the deep neural network based at least in part on the loss function.

9. A method according to claim 8 , wherein the deep neural network is formatted as an auto-encoder representation.

10. A method according to claim 8 , wherein the sequence of training images is a temporal sequence and the particular training image is a middle image of the temporal sequence.

11. A method according to claim 8 , wherein (a) a sequence of pose information is provided to the deep neural network, (b) each instance of the sequence of pose information corresponds to a training image of the sequence of training images, and (c) the one or more features are identified based at least in part on motion indicated by the sequence of pose information.

12. A method according to claim 7 , wherein the routing and navigation system performs one or more localization determinations based on information stored in the feature map layer and at least one of (a) performs one or more route planning decisions, (b) operates a vehicle based on the one or more localization determinations, or (c) both.

13. A method according to claim 7 , wherein the image is captured at nighttime.

14. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising program code instructions configured to:
operate a trained deep neural network to:
receive an image captured under low illumination level conditions and pose data corresponding to the image;
identify a feature within the image;
analyze the image to extract feature information corresponding to the feature from the image;
determine, based on the image and the pose data, geo-location information corresponding to the feature; and
append the geo-location information to the feature information;

store the feature information as part of a feature map layer of a digital map; and
provide at least a portion of the digital map to a routing and navigation system.

15. A computer program product according to claim 14 , the computer-executable program code instructions comprising program code instructions configured to train the deep neural network by:
receiving, by the deep neural network, a sequence of training images, the sequence of training images captured under particular conditions;
analyzing, by the deep neural network, a portion of the sequence of training images, the sequence of training images comprising a particular training image and the portion of the sequence of training images comprising two or more training imag

Assignees

• Here Global Bv

Inventors

• Viswanathan Anirudh

Classifications

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title