Computing high-resolution depth images using machine learning techniques

Patent metadata

Field	Value
Publication number	US-10748247-B2
Application number	US-201715854680-A
Country	US
Kind code	B2
Filing date	Dec 26, 2017
Priority date	Dec 26, 2017
Publication date	Aug 18, 2020
Grant date	Aug 18, 2020

Abstract

A system trains a machine learning model to generate a high-resolution depth image. During a training phase, the system generates an accurate three dimensional reconstruction of a training scene such that the machine learning model is iteratively trained to minimize an error between the higher resolution depth image and the depth information in the accurate three dimensional reconstruction. During a real-time phase, the system applies the trained machine learning model to images captured from a scene of interest and generates a higher resolution depth image with higher accuracy. Thus, the higher resolution depth image can be subsequently used to solve computer vision problems.

First claim

What is claimed is:

1. A method comprising:
receiving a low resolution depth image captured from a scene, the low resolution depth image captured at an initial resolution;
receiving one or more color images captured from the scene;
selecting, based on an object identified in the scene, a machine learning model corresponding to a scene domain for the one or more color images;
applying the machine learning model to the low resolution depth image and the one or more color images,
wherein the machine learning model is trained using an input depth image captured by a depth sensor, input color information derived from a set of color images, and known depth information, and
wherein the machine learning model is trained to generate a high-resolution depth image from the low-resolution depth image and the one or more color images; and

obtaining from the model an upsampled depth image having an upsampled resolution higher than the initial resolution of the low resolution depth image.

2. The method of claim 1 , wherein applying the machine learning model comprises extracting features from the one or more color images and from the low resolution depth image, and providing the extracted features as inputs to the machine learning model.

3. The method of claim 1 , wherein the machine learning model is trained on ground truth values comprising depth information derived from a three dimensional reconstruction of a training scene.

4. The method of claim 1 , wherein the upsampled resolution of the upsampled depth image matches a resolution of the one or more color images captured from the scene.

5. The method of claim 1 , wherein the machine learning model is further trained to generate one or more altered color images each having a resolution that is equal to or greater than a resolution of the received one or more color images.

6. The method of claim 1 , wherein applying the machine learning model comprises:
selecting the machine learning model from a plurality of machine learning models, where the selected machine learning model is selected based on having been trained on training data captured from a training scene corresponding to the identified scene domain.

7. The method of claim 1 , further comprising:
transforming at least one of the received low resolution depth image and the received one or more color images to align together based on relative positions of sensors that captured each of the low resolution depth image and the one or more color images.

8. A non-transitory computer readable medium comprising computer code that, when executed by a processor of a computer, causes the processor to:
receive a low resolution depth image captured from a scene, the low resolution depth image captured at an initial resolution;
receive one or more color images captured from the scene;
select, based on an object identified in the scene, a machine learning model corresponding to a scene domain for the one or more color images;
apply the machine learning model to the low resolution depth image and the one or more color images,
wherein the machine learning model is trained using an input depth image captured by a depth sensor, input color information derived from a set of color images, and known depth information, and
wherein the machine learning model is trained to generate a high-resolution depth image from the low-resolution depth image and the one or more color images; and

obtain from the model an upsampled depth image having an upsampled resolution higher than the initial resolution of the low resolution depth image.

9. The non-transitory computer readable medium of claim 8 , wherein the computer code to apply the machine learning model further comprises computer code that when executed by the processor causes the processor to extract features from the one or more color images and from the low resolution depth image, and provide the extracted features as inputs to the machine learning model.

10. The non-transitory computer readable medium of claim 8 , wherein the machine learning model is trained on ground truth values comprising depth information derived from a three dimensional reconstruction of a training scene.

11. The non-transitory computer readable medium of claim 8 , wherein the upsampled resolution of the upsampled depth image matches a resolution of the one or more color images captured from the scene.

12. The non-transitory computer readable medium of claim 8 , further comprising computer code that, when executed by a processor of a computer, causes the processor to:
obtain one or more altered color images each having a resolution that is equal to or greater than a resolution of the received one or more color images.

13. The non-transitory computer readable medium of claim 8 , where the computer code to apply the machine learning model further comprises computer code that when executed by the processor causes the processor to:
select the machine learning model from a plurality of machine learning models, where the selected machine learning model is selected based on having been trained on training data captured from a training scene corresponding to the identified scene domain.

14. A method comprising:
generating a plurality of examples for a training data set, where generating each example in the training data set comprises:
receiving a low-resolution depth image and one or more color images captured from a scene, where the low resolution depth image and the one or more color images were captured from a camera position in the scene,
receiving three dimensional data captured that describes the scene,
generating a three-dimensional reconstruction of the scene based on the three dimensional data, and
generating a high-resolution depth image of the scene from the camera position based on the three-dimensional reconstruction of the scene, the high-resolution depth image having a higher resolution than the low-resolution depth image;

selecting, based on an object identified in the scene, a machine learning model corresponding to a scene domain; and
training the machine learning model using the training data set, where, for each example in the training data set, the inputs to the machine learning model comprise features extracted from the received low-resolution depth image and features extracted from the received one or more color images, and where the outputs of the machine learning model comprise the generated high-resolution depth image.

15. The method of claim 14 , wherein training the machine learning model comprises, iteratively, for at least a plurality of the example of the training data set:
determining an error between an upsampled depth image generated by the machine learning model for the inputs of the example and the high resolution depth information of the example; and
tuning one or more parameters of the machine learning model to reduce the determined error.

16. The method of claim 14 , wherein the plurality of examples are selected from the training data set based on being captured from a particular scene domain.

17. The method of claim 14 , wherein the three dimensional data is captured by light detection and ranging (LIDAR) sensors, and wherein generating the three dimensional reconstruction comprises processing the three dimensional data using simultaneous localization and mapping (SLAM) algorithms.

18. The method of claim 14 , wherein generating the high-resolution depth image of the scene from the camera position comprises:
for each pixel of the high-resolution depth image, determining a depth from the camera posit

Assignees

• Facebook Inc

Inventors

• Paluri Balmanohar

Classifications

• G06T3/4046 Primary

  using neural networks · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/803

  of input or preprocessed data · CPC title

• G06V20/00

  Scenes; Scene-specific elements (control of digital cameras H04N23/60) · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title