Home filling method using estimated spatio-temporal background information, and recording medium and apparatus for performing the same

Patent metadata

Field	Value
Publication number	US-10497166-B2
Application number	US-201715710442-A
Country	US
Kind code	B2
Filing date	Sep 20, 2017
Priority date	Jun 9, 2017
Publication date	Dec 3, 2019
Grant date	Dec 3, 2019

Abstract

Provided are a hole filling method using estimated spatio-temporal background information and a recording medium and apparatus for performing the same. The present disclosure is providing a hole filling method using estimated spatio-temporal background information, the hole filling method being capable of estimating temporal background information, estimating spatial background information, and merging the temporal background information and the spatial background information to perform a hole filling process, and a recording medium and apparatus for performing the same.

First claim

What is claimed is:

1. A hole filling method using estimated spatio-temporal background information, wherein each frame of an input image includes a color image and a depth image and is divided into non-overlapping patches, the hole filling method comprising:
a temporal background information estimation step of extracting a background codebook for each of non-overlapping patches by using the color image and the depth image per each frame of the input image, and estimating a temporal-background color image and a temporal-background depth image by using the background codebook which is extracted;
a three-dimensional (3D) warping step of performing a 3D warping action on the color image and the depth image per each frame of the input image to acquire a color image of a virtual viewpoint and a depth image of the virtual viewpoint, and performing the 3D warping action on the temporal-background-color image and the temporal-background-depth image to acquire a temporal- background-color image of the virtual viewpoint and a temporal-background-depth image of the virtual viewpoint;
a ghost removal step of removing a ghost phenomenon from the color image of the virtual view point and the depth image of the virtual viewpoint and removing the ghost phenomenon from the temporal-background-color image of the virtual view point and the temporal-background-depth image of the virtual viewpoint;
a spatial background information estimation step of estimating spatial background information in each hidden region of the depth image of the virtual viewpoint from which the ghost phenomenon is removed;
a primary hole filling step of acquiring a spatio-temporal-background-color image of the virtual viewpoint and a spatio-temporal-background-depth image of the virtual viewpoint by using similarity between the temporal-background-depth image of the virtual viewpoint and the spatial-background-depth images of the virtual viewpoint to perform a primary hole filling action; and
a secondary hole filling step of performing an in-painting action based on a priority function including a depth term to perform a hole filling action on remaining holes.

2. The hole filling method of claim 1 , wherein the temporal background information estimation step comprises:
generating a codebook for each of the non-overlapping patches by using the color image and the depth image per each of the frames of the input image;
separating the background codebook from the codebook which is generated; and
estimating temporal background information by using the background codebook which is separated.

3. The hole filling method of claim 2 , wherein the step of generating the codebook comprises finding a codeword corresponding to each of the non-overlapping patches of the input frame from the codebook of each of the non-overlapping patches by using a color distortion, a luminance similarity, and a depth similarity.

4. The hole filling method of claim 1 , wherein the ghost removal step comprises:
obtaining a binary image from the color image of the virtual viewpoint or the depth image of the virtual viewpoint;
generating a boundary mask by using the binary image;
labeling each hidden boundary region by using the boundary mask and obtaining position information of boundary pixels of a foreground region and a background region;
defining a rectangle including each of the labeled hidden boundary regions;
subdividing the rectangle into non-overlapping child rectangles;
dividing depth information of each of the child rectangles into a low-depth cluster and a high-depth cluster; and
removing the ghost phenomenon from a background boundary region of each of the child rectangles by using a minimum depth value of the high-depth cluster as a threshold value.

5. The hole filling method of claim 1 , wherein the spatial background information estimation step comprises:
obtaining a binary mask from the depth image of the virtual viewpoint from which the ghost phenomenon is removed;
labeling each of the hidden regions by using the binary mask and obtaining position information of each of the hidden regions;
defining a rectangle including each of the labeled hidden regions;
subdividing the rectangle into non-overlapping child rectangles;
dividing depth information of each of the child rectangles into a low-depth cluster and a high-depth cluster; and
estimating a lower limit and an upper limit of the spatial-background-depth images of the virtual viewpoint by using minimum and maximum values of depth information of the low-depth cluster.

6. The hole filling method of claim 1 , wherein the primary hole filling step comprises:
comparing the similarity between the temporal-background-depth image of the virtual viewpoint and the spatial-background-depth images of the virtual viewpoint, updating the spatio-temporal-background-depth image of the virtual viewpoint with the temporal-background-depth image of the virtual viewpoint when the temporal-background-depth image of the virtual viewpoint is between a lower limit and an upper limit of the spatial-background-depth images, and otherwise updating the spatio-temporal-background-depth image of the virtual viewpoint with the depth image of the virtual viewpoint from which the ghost phenomenon is removed; and
comparing the similarity between the temporal-background-depth image of the virtual viewpoint and the spatial-background-depth images of the virtual viewpoint, updating the spatio-temporal-background-color image of the virtual viewpoint with the temporal-background-color image of the virtual viewpoint when the temporal-background-depth image of the virtual viewpoint is between the lower limit and the upper limit of the spatial-background-depth images, and otherwise updating the spatio-temporal-background-color image of the virtual viewpoint with the color image of the virtual viewpoint from which the ghost phenomenon is removed.

7. The hole filling method of claim 1 , wherein the secondary hole filling step comprises:
setting the non-overlapping patches centering on each hidden boundary region pixel of a spatio-temporal color image of the virtual viewpoint;
determining a priority of each of the non-overlapping patches by using a priority function including a reliability term, a data term, and a depth term; and
performing an in-painting action on each of the hidden boundary region pixels according to the determined priority to perform a hole filling action on the remaining holes.

8. A non-transitory computer-readable recording medium having recorded thereon a computer program for performing the hole filling method of any one of claims 1 to 7 .

9. A hole filling apparatus using estimated spatio-temporal background information, wherein each frame of an input image includes a color image and a depth image and is divided into non-overlapping patches, the hole filling apparatus comprising:
a temporal background information estimation unit extracting a background codebook for each of non-overlapping patches by using the color image and the depth image per each frame of the input image, and estimating a temporal-background-color image and a temporal-background-depth image by using the background codebook which is extracted;
a three-dimensional (3D) warping unit performing a 3D warping action on the color image and the depth image to acquire a color image of a virtual viewpoint and the depth image of the virtual viewpoint, and perform a 3D warping action on the temporal-background-color image and the temporal-background-depth image to acquire a temporal-background-color image of the virtual viewpoint and a temporal-background-depth image of the virtual viewpoint;
a ghost removal unit removing a ghost phenomenon from the color image of the virtual v

Assignees

• Found Of Soongsil University—Industry Cooperation
• Foundation Soongsil Univ Industry Cooperation

Inventors

• Hong Min-Cheol
• Kim Beomsu
• Nguyen Tien-Daht

Classifications

• G06T7/90

  Determination of colour characteristics · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• H04N13/111

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T15/205

  Image-based rendering · CPC title