Adversarial scene adaptation for camera pose regression

Patent metadata

Field	Value
Publication number	US-11600017-B2
Application number	US-202016861838-A
Country	US
Kind code	B2
Filing date	Apr 29, 2020
Priority date	Apr 29, 2020
Publication date	Mar 7, 2023
Grant date	Mar 7, 2023

Abstract

A pose estimation training system includes: a first model configured to generate a first 6 degrees of freedom (DoF) pose of a first camera that captured a first image from a first domain; a second model configured to generate a second 6 DoF pose of a second camera that captured a second image from a second domain, where the second domain is different than the first domain; a discriminator module configured to, based on first and second outputs from the first and second encoder modules, generate a discriminator output indicative of whether the first and second images are from the same domain; and a training control module configured to, based on the discriminator output, selectively adjust at least one weight value shared by the first model and the second model.

First claim

What is claimed is:

1. A training system, comprising:
a first model including:
a first encoder module configured to generate a first vector representation based on a first image from a first domain using first weight values;
a first localizer module configured to generate a second vector representation based on the first vector representation; and
a first regressor module configured to generate a first 6 degrees of freedom (DoF) pose of a first camera that captured the first image based on the second vector representation using second weight values;

a second model including:
a second encoder module configured to generate a third vector representation based on a second image from a second domain using the first weight values,
wherein the second domain is different than the first domain;
a second localizer module configured to generate a fourth vector representation based on the third vector representation; and
a second regressor module configured to generate a second 6 DoF pose of a second camera that captured the second image based on the fourth vector representation and using the second weight values;

a discriminator module configured to, based on first and second outputs from the first and second encoder modules, generate a discriminator output indicative of whether the first and second images are from the same domain; and
a training control module configured to, based on the discriminator output, selectively adjust at least one of:
at least one of the first weight values; and
at least one of the second weight values.

2. The training system of claim 1 wherein the training control module is configured to adjust at least one of the first weight values until the discriminator output indicates that the first and second images are from the same domain.

3. The training system of claim 1 wherein the training control module is configured to input a different first image to the first model in response to the discriminator output indicating that the first and second images are from the same domain.

4. The training system of claim 1 wherein the training control module is configured to input a different second image to the second model in response to the discriminator output indicating that the first and second images are from the same domain.

5. The training system of claim 1 wherein the training control module is configured to, in response to the discriminator output indicating that the first and second images are from the same domain:
apply a geometric transformation to the first image, thereby generating a geometrically transformed image; and
input the geometrically transformed image to the first model.

6. The training system of claim 5 wherein the geometric transformation includes a rotation of the image by a predetermined angle.

7. The training system of claim 5 wherein the geometric transformation includes one of translation, affine warping, and distortion.

8. The training system of claim 1 wherein the training control module is further configured to store in memory the first model, the first weight values, and the second weight values.

9. The training system of claim 1 wherein:
the first localizer module includes at least one fully connected layers configured to map the first vector representation to the second vector representation; and
the second localizer module includes the at least one fully connected layers configured to map the third vector representation to the fourth vector representation.

10. The training system of claim 1 wherein the first and second regressor modules each include at least two separate connected layers.

11. The training system of claim 1 wherein the first and second encoder modules each include an encoder and a fully connected layer localizer.

12. A vehicle comprising:
at least one propulsion device;
memory including:
the first model of claim 1 ; and
the first and second weight values;

a camera configured to capture an image;
a pose module configured to determine a 6 DoF pose of the camera based on the image and using the first model and the first and second weight values; and
a control module configured to actuate the at least one propulsion device based on the 6 DoF pose of the camera.

13. The training system of claim 1 wherein the first and second DoF poses are absolute 6 DoF poses.

14. The training system of claim 1 wherein the first and second DoF poses are relative 6 DoF poses.

15. The training system of claim 1 wherein the training control module is configured to adjust at least one of the first weight values based on minimizing a loss of the discriminator module.

16. The training system of claim 1 wherein the training control module is configured to adjust at least one of the second weight values based on minimizing a loss of the first and second regressor modules.

17. The training system of claim 1 wherein:
the first 6 DoF includes:
a first three dimensional position of the first camera that captured the first image; and
a first three dimensional orientation of the first camera that captured the first image; and

the second 6 DoF includes:
a second three dimensional position of the second camera that captured the second image; and
a second three dimensional orientation of the second camera that captured the second image.

18. A training method, comprising:
by a first model:
generating a first vector representation based on a first image from a first domain using first weight values;
generating a second vector representation based on the first vector representation; and
generating a first 6 degrees of freedom (DoF) pose of a first camera that captured the first image based on the second vector representation using second weight values;

by a second model:
generating a third vector representation based on a second image from a second domain using the first weight values,
wherein the second domain is different than the first domain;
generating a fourth vector representation based on the third vector representation; and
generating a second 6 DoF pose of a second camera that captured the second image based on the fourth vector representation and using the second weight values;

based on first and second outputs from the first and second models, generating a discriminator output indicative of whether the first and second images are from the same domain; and
based on the discriminator output, selectively adjusting at least one of:
at least one of the first weight values; and
at least one of the second weight values.

19. A non-transitory computer readable medium comprising code that, when executed by one or more processors, performs a training method, comprising:
by a first model:
generating a first vector representation based on a first image from a first domain using first weight values;
generating a second vector representation based on the first vector representation; and
generating a first 6 degrees of freedom (DoF) pose of a first camera that captured the first image based on the second vector representation using second weight values;

by a second model:
generating a third vector representation based on a second image from a second domain using the first weight values,
wherein the second domain is different than the first domain;
generating a fourth vector representation based on the third vector representation; and
generating a second 6 DoF pose of a second camera that captured the second image based on the fourth vector representation and using the second weight values;

based on first and second outpu

Assignees

• Naver Corp
• Naver Labs Corp

Inventors

• Chidlovskii Boris

Classifications

• G06N20/20 Primary

  Ensemble learning · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06T3/60

  Rotation of whole images or parts thereof · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T7/75 Primary

  involving models · CPC title