Image inpainting method and electronic device

Patent metadata

Field	Value
Publication number	US-12283029-B2
Application number	US-202117907849-A
Country	US
Kind code	B2
Filing date	Mar 12, 2021
Priority date	Mar 12, 2020
Publication date	Apr 22, 2025
Grant date	Apr 22, 2025

Abstract

An image inpainting method includes: acquiring an image to be inpainted based on depth information and texture information of a reference image, the image to be inpainted including at least one region to be inpainted; determining at least one reference block matching the at least one region to be inpainted respectively in the reference image; and inpainting the at least one region to be inpainted by using the at least one reference block to obtain a composite image.

First claim

What is claimed is:

1. An image inpainting method, comprising:
acquiring an image to be inpainted based on both depth information and texture information of a reference image, the image to be inpainted including at least one region to be inpainted; wherein each region to be inpainted includes at least one pixel point to be inpainted with a pixel value of zero;
determining at least one reference block matching the at least one region to be inpainted respectively in the reference image; and
inpainting the at least one region to be inpainted according to the at least one reference block respectively to acquire a composite image;
wherein determining the at least one reference block matching the at least one region to be inpainted respectively in the reference image, includes:
determining priorities of all pixel points to be inpainted in a region to be inpainted;
acquiring a target pixel point with a highest priority in all the pixel points to be inpainted;
acquiring a first preset region in the region to be inpainted with the target pixel point as a center; and
matching the first preset region with regions of the reference image, and using a region of the reference image that matches the first preset region as a reference block;
wherein matching the first preset region with the regions of the reference image, and using the region of the reference image that matches the first preset region as the reference block, includes:
acquiring a first reference pixel point corresponding to the target pixel point in the reference image;
acquiring a second preset region of the reference image with the first reference pixel point as a center;
determining a first offset matrix according to the first preset region and the second preset region;
acquiring, for all second reference pixel points in the reference image, third preset regions of the reference image each with a second reference pixel point as a center; the second reference pixel point being any pixel point in the reference image;
determining a second offset matrix according to the first preset region and each third preset region;
determining an offset factor according to the first offset matrix and the second offset matrix;
acquiring a grayscale similarity factor between the second reference pixel point and the target pixel point;
determining a similarity between the first preset region and the third preset region according to the offset factor and the grayscale similarity factor; and
among the third preset regions determined according to all the second reference pixel points, using a preset region with a smallest similarity as the reference block.

2. The method according to claim 1 , wherein determining the priorities of all the pixel points to be inpainted in the region to be inpainted, includes:
acquiring, for each pixel point to be inpainted in the region to be inpainted, a structure factor of the pixel point to be inpainted according to both a unit normal vector and a gradient value of the pixel point to be inpainted;
determining a direction factor of the pixel point to be inpainted according to the pixel point to be inpainted and neighborhood pixel points adjacent to the pixel point to be inpainted; and
determining a priority of the pixel point to be inpainted according to the structure factor and the direction factor.

3. The method according to claim 1 , wherein after acquiring the at least one region to be inpainted of the image to be inpainted, and before determining the priorities of all the pixel points to be inpainted in the region to be inpainted, the method further comprises:
acquiring a plurality of pixel points in the region to be inpainted;
acquiring, for each pixel point of the plurality of pixel points, neighborhood pixel points of each pixel point;
if a number of pixel points not belonging to the region to be inpainted in the neighborhood pixel points of each pixel point is greater than a preset threshold, inpainting the pixel point according to the pixel points not belonging to the region to be inpainted; and
determining at least one pixel point not inpainted in the plurality of pixel points as the at least one pixel point to be inpainted.

4. A non-transitory computer-readable storage medium having stored thereon a computer program that, when executed by a processor, implements steps in the image inpainting method according to claim 1 .

5. The method according to claim 1 , wherein acquiring the image to be inpainted based on both the depth information and the texture information of the reference image, includes:
synthesizing the image to be inpainted, based on a depth-image-based rendering (DIBR) algorithm, according to both the depth information and the texture information of the reference image.

6. The method according to claim 1 , wherein determining the priorities of all the pixel points to be inpainted in the region to be inpainted, includes:
setting priorities of a part of pixel points to be inpainted at edge positions of the region to be inpainted to be low; and
setting priorities of another part of pixel points to be inpainted at center positions of the region to be inpainted to be high.

7. The method according to claim 1 , wherein the at least one region to be inpainted includes a plurality of regions to be inpainted, and the at least one reference block includes a plurality of reference blocks;
inpainting the at least one region to be inpainted according to the at least one reference block respectively to acquire the composite image, includes:
inpainting the plurality of regions to be inpainted sequentially according to the plurality of reference blocks respectively to acquire the composite image; or
inpainting the plurality of regions to be inpainted in parallel according to the plurality of reference blocks respectively to acquire the composite image.

8. An electronic device, comprising: a memory, a processor and a computer program stored on the memory and run on the processor, wherein the computer program, when executed by the processor, implements:
acquiring an image to be inpainted based on both depth information and texture information of a reference image, the image to be inpainted including at least one region to be inpainted; wherein each region to be inpainted includes at least one pixel point to be inpainted with a pixel value of zero;
determining at least one reference block matching the at least one region to be inpainted respectively in the reference image; and
inpainting the at least one region to be inpainted according to the at least one reference block respectively to acquire a composite image;
wherein the computer program, when executed by the processor, further implements;
determining priorities of all pixel points to be inpainted in a region to be inpainted;
acquiring a target pixel point with a highest priority in all the pixel points to be inpainted;
acquiring a first preset region in the region to be inpainted with the target pixel point as a center; and
matching the first preset region with regions of the reference image, and using a region of the reference image that matches the first preset region as a reference block;
wherein the computer program, when executed by the processor, further implements;
acquiring a first reference pixel point corresponding to the target pixel point in the reference image;
acquiring a second preset region of the reference image with the first reference pixel point as a center;
determining a first offset matrix according to the first preset region and the second preset region;
acquiring, for all second reference pixel points in the reference image, third preset regions of the reference image each with a second reference pixel point as a center; the second reference pixel points

Assignees

• Boe Technology Group Co Ltd
• Univ Beijing

Inventors

• Sun Yan
• Tong Yunhe
• Shi Tiankuo
• Chen Xue
• Xi Yanhui
• Bi Yuxin
• Wang Anjie
• Tan Songchao
• Zhang Xiaomang

Classifications

• G06T7/40

  Analysis of texture (depth or shape recovery from texture G06T7/529) · CPC title

• G06V10/761 Primary

  Proximity, similarity or dissimilarity measures · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T7/50

  Depth or shape recovery · CPC title

• G06T5/77 Primary

  Retouching; Inpainting; Scratch removal · CPC title