Camera pose estimation using obfuscated features

Patent metadata

Field	Value
Publication number	US-10839556-B2
Application number	US-201816168601-A
Country	US
Kind code	B2
Filing date	Oct 23, 2018
Priority date	Oct 23, 2018
Publication date	Nov 17, 2020
Grant date	Nov 17, 2020

Abstract

A method for estimating a camera pose includes recognizing a three-dimensional (3D) map representing a physical environment, the 3D map including 3D map features defined as 3D points. An obfuscated image representation is received, the representation derived from an original unobfuscated image of the physical environment captured by a camera. The representation includes a plurality of obfuscated features, each including (i) a two-dimensional (2D) line that passes through a 2D point in the original unobfuscated image at which an image feature was detected, and (ii) a feature descriptor that describes the image feature associated with the 2D point that the 2D line of the obfuscated feature passes through. Correspondences are determined between the obfuscated features and the 3D map features of the 3D map of the physical environment. Based on the determined correspondences, a six degree of freedom pose of the camera in the physical environment is estimated.

First claim

The invention claimed is:

1. A method for estimating a camera pose, comprising:
recognizing a three-dimensional (3D) map representing a physical environment, the 3D map including a plurality of 3D map features defined at a plurality of 3D points distributed throughout the 3D map;
receiving an obfuscated image representation derived from an original unobfuscated image of the physical environment captured by a camera, the obfuscated image representation including a plurality of obfuscated features, and each obfuscated feature including (i) a two-dimensional (2D) line that passes through a 2D point in the original unobfuscated image at which an image feature was detected, and (ii) a feature descriptor associated with the 2D point that describes the image feature that the 2D line of the obfuscated feature passes through;
determining correspondences between the obfuscated features in the obfuscated image representation and the plurality of 3D map features in the 3D map; and
based on the determined correspondences, estimating a six degree of freedom pose of the camera in the physical environment.

2. The method of claim 1 , where determining the correspondences includes identifying a set of obfuscated features having feature descriptors that match feature descriptors of 3D map features in the 3D map, such that the 2D line of each obfuscated feature in the set corresponds to a 3D point associated with a 3D map feature giving a set of 2D line to 3D point correspondences.

3. The method of claim 2 , where, based on the determined 2D line to 3D point correspondences, the pose of the camera is estimated using a random sample consensus (RANSAC) framework and a minimal or non-minimal solver to calculate a plurality of candidate camera poses from subsets of the determined 2D line to 3D point correspondences.

4. The method of claim 3 , where estimating the pose of the camera includes:
identifying a subset of the determined 2D line to 3D point correspondences; and
identifying one or more candidate camera poses consistent with the subset of the determined 2D line to 3D point correspondences.

5. The method of claim 4 , where estimating the pose of the camera further includes identifying additional subsets of the determined 2D line to 3D point correspondences, identifying one or more additional candidate camera poses for each of the additional subsets of 2D line to 3D point correspondences, and identifying a best overall camera pose from the candidate camera poses.

6. The method of claim 3 , where each subset of correspondences includes six 2D line to 3D point correspondences, and each set of candidate camera poses includes sixteen candidate camera poses.

7. The method of claim 3 , where each subset of correspondences includes six 2D line to 3D point correspondences, each 2D line is back-projected as a 3D plane in a camera coordinate system, and a minimal solver calculates a set of eight candidate camera poses by solving an algorithm that outputs pose solutions based on 3D plane to 3D point correspondences.

8. The method of claim 3 , where each subset of correspondences includes twelve 2D line to 3D point correspondences, and each set of candidate camera poses includes a single candidate camera pose.

9. The method of claim 3 , where the obfuscated image representation includes an indication of a gravity vector detected when the original unobfuscated image was captured, each subset of correspondences includes five 2D line to 3D point correspondences, and each set of candidate camera poses includes a single candidate camera pose calculated based on the gravity vector.

10. The method of claim 1 , where a feature descriptor in the obfuscated image representation is a permuted feature descriptor that has been modified with a permutation randomly-selected from a known set of predefined permutations, and the method further comprises applying each permutation of the known set of predefined permutations to the permuted feature descriptor to create a plurality of permuted descriptor copies, searching for correspondences between each permuted descriptor copy and the plurality of 3D map features, and determining a correspondence between the feature descriptor and a 3D map feature if a permuted descriptor copy has at least a threshold similarity to the 3D map feature.

11. A method for estimating a camera pose, comprising:
via a camera, capturing an image of a physical environment;
detecting a plurality of image features at two-dimensional (2D) pixel locations within the image of the physical environment;
generating a plurality of obfuscated features corresponding to the plurality of image features, each obfuscated feature including (i) a 2D line passing through the 2D pixel location of a corresponding image feature, and (ii) a feature descriptor that describes the corresponding image feature associated with the 2D pixel location that the 2D line of the obfuscated feature passes through;
transmitting an obfuscated image representation to a remote device, the obfuscated image representation including the plurality of obfuscated features, the obfuscated image representation not including the 2D pixel locations of the plurality of image features; and
receiving, from the remote device, a six degree of freedom pose of the camera within the physical environment estimated by the remote device based on correspondences between the obfuscated features and a plurality of three-dimensional (3D) map features defined at a plurality of 3D points within a 3D map of the physical environment previously acquired by the remote device.

12. The method of claim 11 , where each image feature detected in the image has a (2D) position, a scale, a one-dimensional (1D) angular orientation, and a feature descriptor, and the obfuscated image representation does not include the 2D position, the scale, and the 1D angular orientation of the image feature.

13. The method of claim 11 , further comprising, for each of the feature descriptors, applying a randomly-selected permutation from a predetermined set of permutations to the feature descriptor prior to transmitting the obfuscated image representation to the remote device.

14. The method of claim 11 , where each 2D line has a randomly assigned direction.

15. A method for estimating a current camera pose, comprising:
recognizing a three-dimensional (3D) map representing a physical environment, the 3D map including a plurality of 3D map features defined at a plurality of 3D points distributed throughout the 3D map;
receiving an obfuscated 3D local map derived from two or more images of the physical environment having been captured by two or more cameras, the obfuscated 3D local map including a plurality of obfuscated features each including (i) a 3D feature representation that replaces a local 3D point of a local 3D image feature in a corresponding unobfuscated 3D local map, the unobfuscated 3D local map including a plurality of local 3D image features at a plurality of local 3D points, each local 3D point having been computed based on feature matching and triangulation of a plurality of 2D image features detected in the two or more images, and (ii) a feature descriptor that describes one of the 2D image features from which the local 3D image feature was triangulated;
determining correspondences between the 3D feature representations in the obfuscated 3D local map and the plurality of 3D points of the plurality of 3D map features in the 3D map of the physical environment; and
based on the determined correspondences, estimating a six degree of freedom pose of the two or more cameras in the physical environment.

16

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Sinha Sudipta Narayan
• Pollefeys Marc Andre Leon
• Kang Sing Bing
• Speciale Pablo Alejandro

Classifications

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06T7/75 Primary

  involving models · CPC title

• G06T2215/16

  Using real world measurements to influence rendering · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T17/10

  Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes · CPC title