Depth superresolution device, depth superresolution method, and program

Patent metadata

Field	Value
Publication number	US-11900622-B2
Application number	US-202017425916-A
Country	US
Kind code	B2
Filing date	Jan 27, 2020
Priority date	Jan 30, 2019
Publication date	Feb 13, 2024
Grant date	Feb 13, 2024

Abstract

Dense depth information can be generated using only a monocular image and sparse depth information.
A depth hyper-resolving apparatus 100 includes: an input data processing unit 22 that outputs a hierarchical input image and hierarchical input depth information by resolution conversion in accordance with a predetermined number of tiers for an input image and input depth information; a depth continuity estimation unit 24 that derives hierarchical estimated depth continuity based on the hierarchical input image; a depth continuity mask deriving unit 26 that outputs a hierarchical depth continuity mask representing values of locations depending on whether a depth is continuous based on the hierarchical input image and the hierarchical estimated depth continuity; and a cost function minimization unit 30 that derives hyper-resolved depth information to minimize a cost function expressed by using the hierarchical input depth information, the hierarchical depth continuity mask, and the hyper-resolved depth information.

First claim

The invention claimed is:

1. A depth hyper-resolving apparatus, comprising:
an input data processing unit configured to output a hierarchical input image and hierarchical input depth information by resolution conversion in accordance with a predetermined number of tiers for an input image and input depth information;
a depth continuity estimation unit configured to derive hierarchical estimated depth continuity based on the hierarchical input image;
a depth continuity mask deriving unit configured to output, based on the hierarchical input image and the hierarchical estimated depth continuity, a hierarchical depth continuity mask representing values of locations depending on whether a depth is continuous; and
a cost function minimization unit configured to derive hyper-resolved depth information to minimize a cost function expressed by using the hierarchical input depth information, the hierarchical depth continuity mask, and the hyper-resolved depth information.

2. The depth hyper-resolving apparatus according to claim 1 , further comprising
an initial value deriving unit configured to output a cost function minimization initial value based on the hierarchical input depth information, wherein
the cost function minimization unit derives the hyper-resolved depth information to minimize the cost function using the cost function minimization initial value.

3. The depth hyper-resolving apparatus according to claim 1 , wherein the cost function minimization unit repeatedly updates a primary variable and a dual variable to minimize the cost function including the primary variable and the dual variable representing the hyper-resolved depth information.

4. The depth hyper-resolving apparatus according to claim 1 , wherein the cost function is expressed including a product obtained by computing the hierarchical depth continuity mask and the hyper-resolved depth information with an operator, and a difference between the hyper-resolved depth information and the hierarchical input depth information.

5. A depth hyper-resolving method, comprising:
outputting, at an input data processing unit, a hierarchical input image and hierarchical input depth information by resolution conversion in accordance with a predetermined number of tiers for an input image and input depth information;
deriving, at a depth continuity estimation unit, hierarchical estimated depth continuity based on the hierarchical input image;
outputting, at a depth continuity mask deriving unit, based on the hierarchical input image and the hierarchical estimated depth continuity, a hierarchical depth continuity mask representing values of locations depending on whether a depth is continuous; and
deriving, at a cost function minimization unit, hyper-resolved depth information to minimize a cost function expressed by using the hierarchical input depth information, the hierarchical depth continuity mask, and the hyper-resolved depth information.

6. The depth hyper-resolving method according to claim 5 , further comprising
outputting, at an initial value deriving unit, a cost function minimization initial value based on the hierarchical input depth information, wherein
deriving at the cost function minimization unit includes deriving the hyper-resolved depth information to minimize the cost function using the cost function minimization initial value.

7. A computer readable recording medium storing a program for causing a computer to function as the depth hyper-resolving apparatus described in claim 1 .

8. The depth hyper-resolving apparatus according to claim 2 , wherein the cost function minimization unit repeatedly updates a primary variable and a dual variable to minimize the cost function including the primary variable and the dual variable representing the hyper-resolved depth information.

9. The depth hyper-resolving apparatus according to claim 2 , wherein the cost function is expressed including a product obtained by computing the hierarchical depth continuity mask and the hyper-resolved depth information with an operator, and a difference between the hyper-resolved depth information and the hierarchical input depth information.

10. The depth hyper-resolving apparatus according to claim 3 , wherein the cost function is expressed including a product obtained by computing the hierarchical depth continuity mask and the hyper-resolved depth information with an operator, and a difference between the hyper-resolved depth information and the hierarchical input depth information.

Assignees

• Nippon Telegraph & Telephone

Inventors

• Yao Yasuhiro
• Ando Shingo
• Kurata Kana
• Niigaki Hitoshi
• Sagata Atsushi

Classifications

• G06T7/50 Primary

  Depth or shape recovery · CPC title

• G06T3/4046

  using neural networks · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title