Image processing method and device, and training method of image processing model and training method thereof

Patent metadata

Field	Value
Publication number	US-12198302-B2
Application number	US-202117775340-A
Country	US
Kind code	B2
Filing date	Jul 15, 2021
Priority date	Jul 15, 2021
Publication date	Jan 14, 2025
Grant date	Jan 14, 2025

Abstract

An image processing method includes: acquiring a first image containing a target object; inputting the first image into an image processing model to obtain a second image, the second image being a mask image of the target object in the first image, a value for each pixel in the second image being in a range of 0 to 1, inclusive ([0, 1]), and the range of 0 to 1, inclusive ([0, 1]) indicating a degree of relation between each pixel in the second image and a pixel in the target object; fusing the first image and a background image according to the second image to obtain a fused image; and providing a first interface and displaying the fused image on the first interface.

First claim

What is claimed is:

1. An image processing method, comprising:
acquiring a first image containing a target object;
inputting the first image into an image processing model to obtain a second image, the second image being a mask image of the target object in the first image, a value for each pixel in the second image being in a range of 0 to 1, inclusive ([0, 1]), and the range of 0 to 1, inclusive ([0, 1]) indicating a degree of relation between each pixel in the second image and a pixel in the target object;
fusing the first image and a background image according to the second image to obtain a fused image; and
providing a first interface and displaying the fused image on the first interface;
wherein the target object is a person figure, and the image processing model is an image processing model trained by using following steps:
inputting a portrait with simple background, an image with complex background and a portrait with complex background into a pre-trained image processing model;
determining an error function of the pre-trained image processing model; and
training the pre-trained image processing model according to the error function to obtain the image processing model,
wherein a truth-value image of the portrait with simple background and a truth-value image of the portrait with complex background are both a mask image of the portrait with simple background, and a truth-value image of the image with complex background is an all-zero image.

2. The image processing method according to claim 1 , wherein acquiring the first image containing the target object includes:
providing a second interface displaying a user input indication;
receiving a first instruction input by a user; and
acquiring the first image containing the target object in response to the first instruction.

3. The image processing method according to claim 2 , wherein after displaying the fused image on the first interface, the image processing method further comprises:
receiving a second instruction input by the user; and
displaying the second interface in response to the second instruction.

4. The image processing method according to claim 1 , wherein after displaying the fused image on the first interface, the image processing method further comprises:
receiving a third instruction input by a user; and
editing the fused image displayed on the first interface by using one or more image editing manners in response to the third instruction.

5. The image processing method according to claim 1 , wherein the fused image, the first image, the second image, and the background image satisfy a following relationship:
I fusion =T fusion1 [T fusion2 ×I person ×I mask +T fusion3 ×I background ×(1− I mask )],

wherein I fusion represents the fused image, I person represents the first image, I background represents the background image, I mask represents the second image, a value of each element of I mask is in a range of 0 to 1, inclusive ([0, 1]), and T fusion1 , T fusion2 and T fusion3 each represent an image processing function.

6. The image processing method according to claim 1 , further comprising:
acquiring computing resource information of a computing device for performing the image processing method; and
adjusting image processing resolution of the image processing model according to the computing resource information.

7. The image processing method according to claim 1 , wherein the portrait with complex background is a composite image of the portrait with simple background and the image with complex background;
before inputting the portrait with simple background, the image with complex background and the portrait with complex background into the pre-trained image processing model, the following steps used to train the image processing model further includes:
fusing the portrait with simple background and the image with complex background according to the mask image of the portrait with simple background to obtain another fused image; and
performing augmentation processing on the another fused image to obtain the portrait with complex background.

8. The image processing method according to claim 1 , wherein the portrait with complex background, the portrait with simple background, and the image with complex background satisfy a following relationship:
I fusion ′=T random1 [T random2 ×I person ′×I mask ′+T random3 ×I background ′×(1− I mask ′)],

wherein I fusion ′ represents the portrait with complex background, I person ′ represents the portrait with simple background, I background ′ represents the image with complex background, I mask ′ represents the mask image of the portrait with simple background, a value of each element of I mask ′ is in a range of 0 to 1, inclusive ([0, 1]), and T random1 , T random2 and T random3 each represent a random image processing function.

9. The image processing method according to claim 1 , wherein the pre-trained image processing model includes a salient object detection model; and the mask image of the portrait with simple background includes a saliency probability map output by the salient object detection model after the portrait with simple background is input into the salient object detection model; or
the pre-trained image processing model is a first U 2 -Net model, and the mask image of the portrait with simple background is a saliency probability map output by a second U 2 -Net model after the portrait with simple background is processed by using the second U 2 -Net model.

10. The image processing method according to claim 1 , wherein determining the error function of the pre-trained image processing model includes:
comparing an output image of the pre-trained image processing model with a corresponding truth-value image to obtain the error function.

11. An image processing device, comprising:
a memory having stored thereon one or more computer programs, and
a processor coupled to the memory, and configured to execute the one or more computer program to implement the image processing method according to claim 1 .

12. A non-transitory computer-readable storage medium having stored computer program instructions thereon, wherein the computer program instructions, when run on a computer, cause the computer to implement the image processing method according to claim 1 .

13. A training method of an image processing model, comprising:
inputting a portrait with simple background, an image with complex background and a portrait with complex background into a pre-trained image processing model;
determining an error function of the pre-trained image processing model; and
training the pre-trained image processing model according to the error function to obtain the image processing model,
wherein a truth-value image of the portrait with simple background and a truth-value image of the portrait with complex background are both a mask image of the portrait with simple background, and a truth-value image of the image with complex background is an all-zero image.

14. The training method according to claim 13 , wherein the portrait with complex background is a composite image of the portrait with simple background and the image with complex background;
before inputting the portrait with simple background, the image with complex background and the portrait with complex background into the pre-trained image processing model, the training method further comprises:
fusing the portrait with simple background and the image with complex background according to the mask image of the portrait with simple background to obtain a f

Assignees

• Boe Technology Group Co Ltd

Inventors

• Duan Ran
• Liu Hanwen
• Lu Yunhua

Classifications

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06T7/194

  involving foreground-background segmentation · CPC title