Use of inpainting techniques for image correction

Patent metadata

Field	Value
Publication number	US-9256926-B2
Application number	US-200913055727-A
Country	US
Kind code	B2
Filing date	Jul 22, 2009
Priority date	Jul 28, 2008
Publication date	Feb 9, 2016
Grant date	Feb 9, 2016

Abstract

A method of processing an image signal comprising image and depth information is provided. The method is configured to perform segmentation on an image based on depth/disparity information present in the image signal comprising said image, and subsequently inpaint background for correction of the errors in the image around the foreground objects into a region that extends beyond the segment boundary of the foreground object and/or inpaint foreground for correction of errors in the image into a region that extends inside the segment boundary of the foreground object. In this way compression and other artifacts may be reduced.

First claim

The invention claimed is:

1. A method for operating an image processor unit for processing an image signal comprising image and depth information, the method comprising the acts of:
in an image processor unit, a processor executing code segments stored in a memory for performing the acts of:
determining and storing in the memory, image and depth information for an image determined from an image signal,
performing foreground/background segmentation on the image based on the depth or disparity component of the image signal, for determining a foreground mask and a background mask for the image,
determining an area around a segmentation transition in the segmented image, wherein the area comprises artifacts,
determining a location of the artifacts with regard to the foreground mask and the background mask, wherein the location of the artifacts within the area is determined as high-frequency variation of pixel data caused by compression;
extending the area into one of the foreground mask and the background mask, based on the location of the artifacts, and
performing one of foreground dependent and background dependent processing on pixel data in the extended area, resulting in a processed image, the dependent processing comprising:
adjusting color information inside the extended area to a boundary of the foreground mask.

2. The method according to claim 1 , wherein the dependent processing is one of: an inpainting technique of temporal inpainting, spatial inpainting, and a spatio-temporal inpainting.

3. The method according to claim 1 , the adjusting color information comprising:
replacing said pixel data in the extended area with pixel data outside the extended area.

4. The method according to claim 1 , said adjusting color information comprising:
blending said pixel data in the extended area with pixel data located outside the extended area.

5. The method according to claim 4 , wherein the blending factor is a function of a distance to the segmentation transition of the extended area.

6. The method according to claim 1 , wherein said adjusting color information comprising:
changing said pixel data in the extended area based on pixel data originating from stereo or multiview image pairs.

7. The method according to claim 1 , wherein said adjusting color information is performed by utilizing foreground color information from the foreground mask to inpaint the pixel data in the extended area.

8. The method according to claim 1 , wherein said adjusting color information is performed by utilizing background color information from the background mask to inpaint the pixel data in area.

9. The method according to claim 1 , wherein said area is derived from data in the bitstream.

10. The method according to claim 9 , wherein said data is a block size used for compression.

11. The method according to claim 9 , wherein said data is a quantization parameter.

12. The method according to claim 9 , wherein said data is a de-blocking filter setting.

13. A processing unit processing an image signal comprising image and depth information, comprising:
a memory; and
a processor in communication with the memory, the memory being configured to control the processor to:
perform foreground/background segmentation on an image comprised in the image signal based on the depth or disparity component of the image signal, for defining a foreground mask and a background mask,
determine an area adjacent to a segmentation transition in the segmented image, wherein the area comprises artifacts,
determine a location of the artifacts with regard to the foreground mask and the background mask, wherein the location of the artifacts within the area is determined as high-frequency variation of pixel data caused by compression;
extend the area into one of the foreground mask and the background mask, based on the location of the artifacts, and
perform one of foreground dependent and background dependent processing on pixel data in the extended area, resulting in a processed image, the dependent processing comprising adjusting color information inside the extended area to a boundary of the foreground mask.

14. A computer program product stored on a non-transitory computer-readable medium comprising software code which when executed on a data-processing apparatus performs the acts of:
in an image processor unit, a processor executing code segments stored in a memory for performing the acts of:
determining and storing in the memory, image and depth information for an image determined from an image signal,
performing foreground/background segmentation on an image comprised in the image signal based on the depth or disparity component of the image signal, resulting in a foreground mask and a background mask;
creating an area around a segmentation transition in the segmented image, wherein the area comprises artifacts,
determining a location of the artifacts with regard to the foreground mask and the background mask, wherein the location of the artifacts within the area is determined as high-frequency variation of pixel data caused by compression;
extending the area into one of the foreground mask and the background mask, based on the location of the artifacts, and
performing one of foreground dependent and background dependent processing on pixel data in the extended area, the dependent processing comprising adjusting color information inside the extended area to a boundary of the foreground mask.

15. The processing unit according to claim 13 wherein the dependent processing is one of: an inpainting technique of temporal inpainting, spatial inpainting, and a spatio-temporal inpainting.

16. The processing unit according to claim 13 the adjusting color information comprising:
replacing said pixel data in the extended area with pixel data outside the extended area.

17. The processing unit according to claim 13 said adjusting color information comprising:
blending said pixel data in the extended area with pixel data located outside the extended area.

18. The processing unit according to claim 17 , wherein the blending factor is a function of a distance to the segmentation transition of the extended area.

19. The processing unit according to claim 13 wherein said adjusting color information comprising:
changing said pixel data in the extended area based on pixel data originating from stereo or multiview image pairs.

20. The processing unit according to claim 13 wherein said adjusting color information is performed by utilizing foreground color information from the foreground mask to inpaint the pixel data in the extended area.

21. The processing unit according to claim 13 wherein said adjusting color information is performed by utilizing background color information from the background mask to inpaint the pixel data in area.

22. The processing unit according to claim 13 wherein said area is derived from data in the bitstream.

23. The processing unit according to claim 22 , wherein said data is a block size used for compression.

24. The processing unit according to claim 22 , wherein said data is a quantization parameter.

25. The processing unit according to claim 22 , wherein said data is a de-blocking filter setting.

Assignees

• Berretty Robert-Paul Mario
• Magalhaes Jose Pedro
• Klein Gunnewiek Reinier Bernardus Maria
• Barenbrug Bart Gerard Bernard
• Shao Ling
• Koninkl Philips Nv

Inventors

• Berretty Robert-Paul Mario
• Magalhaes Jose Pedro
• Klein Gunnewiek Reinier Bernardus Maria
• Barenbrug Bart Gerard Bernard
• Shao Ling

Classifications

• H04N19/86

  involving reduction of coding artifacts, e.g. of blockiness · CPC title

• H04N19/17

  the unit being an image region, e.g. an object · CPC title

• G06T5/005 Primary

  Physics · mapped topic

• H04N19/20

  using video object coding · CPC title

• H04N13/00

  Stereoscopic video systems; Multi-view video systems; Details thereof · CPC title