Method and apparatus for filling holes in naked-eye 3D multi-viewpoint image, and electronic device for performing the method

Patent metadata

Field	Value
Publication number	US-11043152-B2
Application number	US-201916438792-A
Country	US
Kind code	B2
Filing date	Jun 12, 2019
Priority date	Jul 3, 2018
Publication date	Jun 22, 2021
Grant date	Jun 22, 2021

Abstract

A method for filling holes in a naked-eye 3D multi-viewpoint image is provided, comprising: sequentially performing a step of mapping, determining and marking row by row, comprising mapping two first pixels adjacent to each other in a row direction among a plurality of first pixels in a reference camera image to two second pixels in a virtual camera image, determining whether there is a hole between the two second pixels, and if so, marking the hole; and sequentially performing a step of scanning, assessing and filling row by row, comprising scanning two second pixels adjacent to each other in a row direction, assessing whether there is a marked hole between the two adjacent second pixels, and filling the marked hole according to at least one non-zero second pixel which is adjacent to the marked hole.

First claim

What is claimed is:

1. A method for filling holes in a naked-eye 3D multi-viewpoint image, comprising:
sequentially performing a step of mapping, determining and marking row by row, the step of mapping, determining and marking comprises mapping two first pixels adjacent to each other in a row direction among a plurality of first pixels in a reference camera image to two second pixels in a virtual camera image, and determining whether there is a hole with a gray scale of zero between the two mapped second pixels, and if so, marking the hole; and
sequentially performing a step of scanning, assessing and filling row by row, the step of scanning, assessing and filling comprises scanning two second pixels adjacent to each other in a row direction, assessing whether there is a marked hole between the two adjacent second pixels, and replacing the marked hole according to at least one non-zero second pixel adjacent to the marked hole,
wherein the step of replacing the marked hole according to at least one non-zero second pixel adjacent to the marked hole comprises:
if the marked hole does not include at least three consecutive vacant pixels, acquiring at least one non-zero second pixel from a pixel array centered on the vacant pixel, calculating a gray gradient of the at least one non-zero second pixel in a direction directing from the vacant pixel to the at least one non-zero second pixel respectively, and replacing the vacant pixel with the non-zero second pixel in the direction in which the gray gradient is minimum;
if the marked hole includes at least three consecutive vacant pixels and the virtual camera image is shifted to the right relative to the reference camera image, sequentially for the vacant pixels from left to right, acquiring at least one non-zero second pixel from a pixel array centered on the vacant pixel, calculating a gray gradient of the at least one non-zero second pixel in a direction directing from the vacant pixel to the at least one non-zero second pixel respectively, and replacing the vacant pixel with the non-zero second pixel in the direction in which the gradient is minimum,
if the marked hole includes at least three consecutive vacant pixels and the virtual camera image is shifted to the left relative to the reference camera image, sequentially for the vacant pixels from right to left, acquiring at least one non-zero second pixel from a pixel array centered on the vacant pixel, calculating a gray gradient of the at least one non-zero second pixel in a direction directing from the vacant pixel to the at least one non-zero second pixel respectively, and replacing the vacant pixel with the non-zero second pixel in the direction in which the gradient is minimum.

2. The method for filling holes in a naked-eye 3D multi-viewpoint image of claim 1 , wherein the step of determining whether there is a hole with a gray scale of zero between the two mapped second pixels comprises:
obtaining a difference between horizontal coordinates of the two second pixels, determining whether the difference is greater than a difference between horizontal coordinates of two adjacent mapped second pixels without a hole between the two adjacent mapped second pixels, and if so, determining that there is a hole between the two mapped second pixels.

3. The method for filling holes in a naked-eye 3D multi-viewpoint image of claim 2 , wherein the step of marking the hole comprises marking the hole with a fixed pixel value.

4. The method for filling holes in a naked-eye 3D multi-viewpoint image of claim 3 , the step of assessing whether there is a marked hole between the two adjacent second pixels, further comprises assessing whether the marked hole includes at least three consecutive vacant pixels.

5. The method for filling holes in a naked-eye 3D multi-viewpoint image of claim 4 , wherein the step of assessing whether the marked hole includes at least three consecutive vacant pixels comprises detecting a pixel value of each of the two adjacent second pixels and vacant pixels between the two adjacent second pixels to assess whether the marked hole includes at least three consecutive vacant pixels.

6. The method for filling holes in a naked-eye 3D multi-viewpoint image of claim 1 , wherein the step of acquiring at least one non-zero second pixel from a pixel array centered on the vacant pixel comprises acquiring four non-zero pixels in different directions of neighborhoods of the vacant pixel from a pixel array centered on the vacant pixel.

7. An electronic device, comprising at least one processor; and
a memory communicatively coupled to the at least one processor; wherein
the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor performs the method of claim 1 .

8. An apparatus for filling holes in a naked-eye 3D multi-viewpoint image, comprising:
a mapping circuit which is configured to sequentially map two first pixels adjacent to each other in a row direction in a reference camera image to two second pixels in a virtual camera image row by row;
a judging and marking circuit which is configured to sequentially determine whether there is a hole with a gray scale of zero between the two mapped second pixels, and if so, mark the hole row by row;
a scanning and assessing circuit which is configured to sequentially scan two adjacent second pixels adjacent to each other in a row direction, assessing whether there is a marked hole between the two adjacent second pixels row by row;
a filling circuit which is configured to replace the marked hole according to at least one non-zero second pixel adjacent to the marked hole; wherein
the mapping circuit, the judging and marking circuit, the scanning and assessing circuit and the filling circuit are sequentially connected,
wherein, if the marked hole does not include at least three consecutive vacant pixels, the filling circuit is configured to acquire at least one non-zero second pixel from a pixel array centered on the vacant pixel, calculate a gray gradient of the at least one non-zero second pixel in a direction directing from the vacant pixel to the at least one non-zero second pixel respectively, and replace the vacant pixel with the non-zero second pixel in the direction in which the gray gradient is minimum,
if the marked hole includes at least three consecutive vacant pixels and the virtual camera image is shifted to the right relative to the reference camera image, the filling circuit is configured, sequentially for the vacant pixels from left to right, to acquire at least one non-zero second pixel from a pixel array centered on the vacant pixel, calculate a gray gradient of the at least one non-zero second pixel in a direction directing from the vacant pixel to the at least one non-zero second pixel respectively, and replace the vacant pixel with the non-zero second pixel in the direction in which the gradient is minimum,
if the marked hole includes at least three consecutive vacant pixels and the virtual camera image is shifted to the left relative to the reference camera image, the filling circuit is configured, sequentially for the vacant pixels from right to left, acquire at least one non-zero second pixel from a pixel array centered on the vacant pixel, calculate a gray gradient of the at least non-zero second pixel in a direction directing from the vacant pixel to the at least one non-zero second pixel respectively, and replace the vacant pixel with the non-zero second pixel in the direction in which the gradient is minimum.

9. The apparatus for filling holes in a naked-eye 3D multi-viewpoint image of claim 8 , the scanning and assessing circuit is further configured to assess wh

Assignees

• Boe Technology Group Co Ltd

Inventors

• Dai Bin
• Zhang Xiaomang
• Xi Yanhui
• Sun Yan
• Gao Yankai
• Bi Yuxin
• Shi Lingyun

Classifications

• G09G3/3291

  in which the data driver supplies a variable data voltage for setting the current through, or the voltage across, the light-emitting elements · CPC title

• H04N13/122 Primary

  Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence) · CPC title

• G09G3/007 Primary

  Use of pixel shift techniques, e.g. by mechanical shift of the physical pixels or by optical shift of the perceived pixels · CPC title

• G09G3/3607

  for displaying colours or for displaying grey scales with a specific pixel layout, e.g. using sub-pixels (display of colours in flat matrix panels other than liquid crystal displays G09G3/2003; grey scales specific for television H04N3/127) · CPC title

• G09G2300/0819

  used for counteracting undesired variations, e.g. feedback or autozeroing · CPC title