Image processing method and electronic device to improve an image compositing effect

Patent metadata

Field	Value
Publication number	US-12175681-B2
Application number	US-202017636542-A
Country	US
Kind code	B2
Filing date	Jul 30, 2020
Priority date	Aug 22, 2019
Publication date	Dec 24, 2024
Grant date	Dec 24, 2024

Abstract

An image processing method includes displaying a first key frame image obtained from a sequence frame video stream; obtaining a first foreground image and a first background image obtained after foreground and background separation processing is performed on the first key frame image; obtaining a second foreground image obtained after foreground image processing is performed on a first target object in the first foreground image; obtaining a third background image obtained after background repair processing is performed on the first background image using a second background image, where the second background image is a background image included in a second key frame image obtained by photographing a target scene before the first key frame image is obtained; and obtaining a first key frame image obtained after foreground and background compositing processing is performed on the second foreground image and the third background image.

First claim

What is claimed is:

1. A method comprising:
photographing, using a camera of an electronic device, a target scene to obtain a second key frame image, wherein the second key frame image comprises a second background image;
photographing, using the camera, the target scene comprising a first target object to obtain a sequence frame video stream;
displaying a first key frame image of the sequence frame video stream by:
displaying n sequence frame images of the sequence frame video stream, wherein n is a positive integer greater than or equal to 2;
obtaining a first original key frame image and n−1 related frame images from the n sequence frame images;
obtaining an original image point energy value of the first original key frame image;
obtaining k front related frame images and k rear related frame images of the first original key frame image from the n−1 related frame images, wherein a value of k is less than or equal to (n−1)÷2; and
performing optimization processing on the original image point energy value using image point energy values of the k front related frame images and the k rear related frame images to obtain an optimized image point energy value of the first original key frame image;

performing foreground and background separation processing on the first key frame image to obtain a first foreground image and a first background image, wherein the first foreground image comprises the first target object;
performing foreground image processing on the first target object to obtain a second foreground image, wherein the second foreground image comprises the first target object;
performing, using the second background image, background repair processing on the first background image to obtain a third background image; and
performing foreground and background compositing processing on the second foreground image and the third background image to obtain a third key frame image.

2. The method of claim 1 , wherein before displaying the first key frame image, the method further comprises:
performing, before obtaining the sequence frame video stream, preview photographing on the target scene to obtain a preview video stream;
displaying the second key frame image from the preview video stream; and
obtaining the second background image separated from the second key frame image.

3. The method of claim 1 , wherein before obtaining the third background image, the image processing method further comprises:
displaying the second key frame image;
performing the foreground and background separation processing on the second key frame image to obtain a fourth background image;
performing preview photographing on the target scene before obtaining the sequence frame video stream to obtain a preview video stream, wherein the preview video stream comprises a fourth key frame image;
separating a fifth background image from the fourth key frame image; and
performing the background repair processing on the fourth background image using the fifth background image to obtain the second background image.

4. The method of claim 1 , wherein before displaying the first key frame image, the method further comprises:
consecutively photographing, using the camera, the target scene to obtain a plurality of consecutive background images before obtaining the sequence frame video stream; and
performing cumulative superimposition processing on the consecutive background images to obtain the second background image.

5. The method of claim 1 , further comprising:
performing definition enhancement processing on the first original key frame image using pixel information of the n−1 related frame images to obtain a second original key frame image; and
setting, as the first key frame image, the second original key frame image.

6. The method of claim 1 , wherein after obtaining the third key frame image, the method further comprises:
performing foreground edge fusion processing on the third key frame image to obtain a fourth key frame image; and
outputting the fourth key frame image.

7. The method of claim 1 , further comprising:
obtaining a changed inter-frame location relationship between the first background image and the second background image based on a feature point correspondence between the first background image and the second background image;
performing perspective transformation on the second background image based on the changed inter-frame location relationship to obtain a transformed second background image; and
stitching the transformed second background image to the first background image to obtain the third background image.

8. The method of claim 7 , further comprising:
obtaining a current-frame posture parameter corresponding to the first background image;
obtaining an initial inter-frame location relationship between the first background image and the second background image based on a previous-frame posture parameter corresponding to the second background image and the current-frame posture parameter;
performing feature extraction and feature matching on the first background image and the second background image to obtain the feature point correspondence; and
optimizing the initial inter-frame location relationship using the feature point correspondence to obtain the changed inter-frame location relationship.

9. The method of claim 8 , further comprising:
obtaining a perspective transformation matrix between the first background image and the second background image using the changed inter-frame location relationship; and
further performing the perspective transformation on the second background image using the perspective transformation matrix to obtain the transformed second background image.

10. The method of claim 1 , further comprising:
obtaining a background object image from the second background image through segmentation;
identifying that the background object image has entered a foreground missing region in the first background image; and
performing, in response to the identifying, background patching processing on the first background image using the background object image.

11. An electronic device comprising:
a camera;
a memory configured to store instructions; and
a processor coupled to the memory and the camera, wherein when executed by the processor, the instructions cause the electronic device to:
photograph, using the camera, a target scene to obtain a second key frame image, wherein the second key frame image comprises a second background image;
photograph, using the camera, the target scene comprising a first target object to obtain a sequence frame video stream;
display a first key frame image of the sequence frame video stream, wherein to display the first key frame image, when executed by the processor, the instructions further cause the electronic device to:
display n sequence frame images of the sequence frame video stream, wherein n is a positive integer greater than or equal to 2;
obtain a first original key frame image and n−1 related frame images from the n sequence frame images;
obtain an original image point energy value of the first original key frame image;
obtain k front related frame images and k rear related frame images of the first original key frame image from the n−1 related frame images, wherein a value of k is less than or equal to (n−1)÷2; and
perform optimization processing on the original image point energy value using image point energy values of the k front related frame images and the k rear related frame images to obtain an optimized image point energy value of the first original key frame image;

perform foreground and background separation processing on the first key frame image to obtain a f

Assignees

• Huawei Tech Co Ltd

Inventors

• Wang Liqiang
• Lu Henghui

Classifications

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T3/4038

  Image mosaicing, e.g. composing plane images from plane sub-images · CPC title

• G06T7/194 Primary

  involving foreground-background segmentation · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title