Learning-Based Camera Pose Estimation From Images of an Environment

Patent metadata

Field	Value
Publication number	US-2019108651-A1
Application number	US-201816137064-A
Country	US
Kind code	A1
Filing date	Sep 20, 2018
Priority date	Oct 6, 2017
Publication date	Apr 11, 2019
Grant date	—

Abstract

A deep neural network (DNN) system learns a map representation for estimating a camera position and orientation (pose). The DNN is trained to learn a map representation corresponding to the environment, defining positions and attributes of structures, trees, walls, vehicles, walls, etc. The DNN system learns a map representation that is versatile and performs well for many different environments (indoor, outdoor, natural, synthetic, etc.). The DNN system receives images of an environment captured by a camera (observations) and outputs an estimated camera pose within the environment. The estimated camera pose is used to perform camera localization, i.e., recover the three-dimensional (3D) position and orientation of a moving camera, which is a fundamental task in computer vision with a wide variety of applications in robot navigation, car localization for autonomous driving, device localization for mobile navigation, and augmented/virtual reality.

First claim

What is claimed is:

1 . A computer-implemented method, comprising:
receiving an input image at a deep neural network (DNN), wherein weights of the DNN define a map representation of an environment and the weights are determined during training using a labeled training dataset including images and corresponding absolute camera poses and relative camera poses; and applying, by the DNN, the weights to the input image to generate an estimated camera pose for capturing the environment to produce the input image.

2 . The computer-implemented method of claim 1 , wherein, during the training, image pairs are input to the DNN and corresponding estimated camera pose pairs are generated by the DNN.

3 . The computer-implemented method of claim 2 , wherein an image pair includes a first image and an additional image in an image sequence, and one or more intervening images may occur between the first image and the additional image.

4 . The computer-implemented method of claim 2 , wherein, during the training, relative estimated camera poses are computed for each estimated camera pose pair.

5 . The computer-implemented method of claim 4 , wherein, during the training, the weights are modified to simultaneously reduce differences between the relative estimated camera poses and the relative camera poses included in the training dataset and differences between the estimated camera poses generated by the DNN and the absolute camera poses included in the training dataset.

6 . The computer-implemented method of claim 1 , wherein a rotation portion of the estimated camera pose is parameterized as a three-dimensional logarithm of a unit quaternion.

7 . The computer-implemented method of claim 1 , further comprising:
receiving visual odometry data corresponding to the input image; and modifying the weights of the DNN to minimize differences between the visual odometry data and a relative camera pose computed using the estimated camera pose and an additional estimated camera pose generated by the DNN.

8 . The computer-implemented method of claim 1 , further comprising:
receiving global position sensor data corresponding to the input image; and modifying the weights to minimize differences between the global position sensor data and the estimated camera pose.

9 . The computer-implemented method of claim 1 , further comprising:
receiving inertial measurement data corresponding to the input image; and modifying the weights to minimize differences between the inertial measurement data and the estimated camera pose.

10 . The computer-implemented method of claim 1 , further comprising post-processing the estimated camera pose using pose graph optimization (PGO) to produce a refined camera pose.

11 . The computer-implemented method of claim 1 , wherein the DNN comprises at least a convolutional neural network layer, followed by a global average pooling layer, followed by a fully-connected layer to output the estimated camera pose.

12 . A system, comprising:
a deep neural network (DNN) configured to:
receive an input image, wherein weights of the DNN define a map representation of an environment and the weights are determined during training using a labeled training dataset including images and corresponding absolute camera poses and relative camera poses; and
apply the weights to the input image to generate an estimated camera pose for capturing the environment to produce the input image.

13 . The system of claim 12 , wherein, during the training, image pairs are input to the DNN and corresponding estimated camera pose pairs are generated by the DNN.

14 . The system of claim 13 , wherein an image pair includes a first image and an additional image in an image sequence, and one or more intervening images may occur between the first image and the additional image.

15 . The system of claim 13 , further comprising a relative pose computation unit configured to compute relative estimated camera poses for each estimated camera pose pair during the training.

16 . The system of claim 15 , further comprising a training loss unit configured to modify the weights during the training to simultaneously reduce differences between the relative estimated camera poses and the relative camera poses included in the training dataset and differences between the estimated camera poses generated by the DNN and the absolute camera poses included in the training dataset.

17 . The system of claim 12 , wherein a rotation portion of the estimated camera pose is parameterized as a three-dimensional logarithm of a unit quaternion.

18 . The system of claim 12 , further comprising a training loss unit configured to:
receive visual odometry data corresponding to the input image; and modify the weights of the DNN to minimize differences between the visual odometry data and a relative camera pose computed using the estimated camera pose and an additional estimated camera pose generated by the DNN.

19 . The system of claim 12 , further comprising a pose graph optimization unit configured to post-process the estimated camera pose using pose graph optimization (PGO) to produce a refined camera pose.

20 . A non-transitory computer-readable media storing computer instructions for estimating camera poses that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving an input image at a deep neural network (DNN), wherein weights of the DNN define a map representation of an environment and the weights are determined during training using a labeled training dataset including images and corresponding absolute camera poses and relative camera poses; and applying, by the DNN, the weights to the input image to generate an estimated camera pose for capturing the environment to produce the input image.

Assignees

• Nvidia Corp

Inventors

• Gu Jinwei
• Brahmbhatt Samarth Manoj
• Kim Kihwan
• Kautz Jan

Classifications

• G06T7/20 Primary

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06T7/80 Primary

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title

• G06V10/955

  using specific electronic processors · CPC title