Video generation method and apparatus, and device and storage medium

Patent metadata

Field	Value
Publication number	US-12526460-B2
Application number	US-202218715068-A
Country	US
Kind code	B2
Filing date	Nov 29, 2022
Priority date	Nov 30, 2021
Publication date	Jan 13, 2026
Grant date	Jan 13, 2026

Abstract

Embodiments of the present disclosure disclose a video generation method and apparatus, a device, and a storage medium. The method includes: obtaining a video frame contained in a video to be processed, where the video to be processed contains a portrait; performing portrait segmentation on the video frame to obtain a portrait image and a background image; adjusting transparency of pixels that meet a defined condition in the portrait image to obtain the adjusted portrait image; processing the background image to obtain an entire background image; fusing the adjusted portrait image and the entire background image to obtain a portrait video frame; and stitching a plurality of portrait video frames to obtain a target video.

First claim

The invention claimed is:

1 . A video generation method, comprising: obtaining a video frame contained in a video to be processed, wherein the video to be processed contains a portrait; performing portrait segmentation on the video frame to obtain a portrait image and a background image; adjusting transparency of pixels that meet a defined condition in the portrait image to obtain the adjusted portrait image; processing the background image to obtain an entire background image; fusing the adjusted portrait image and the entire background image to obtain a portrait video frame; and stitching a plurality of portrait video frames to obtain a target video;
wherein adjusting the transparency of the pixels that meet the defined condition in the portrait image to obtain the adjusted portrait image comprises: making at least one copy of the portrait image to obtain at least one portrait image copy; rotating the at least one portrait image copy about a coordinate axis of a three-dimensional space by a set angle to obtain the rotated portrait image copy, wherein the portrait image and the at least one rotated portrait image copy form a portrait image group; and adjusting transparency of pixels that meet the defined condition in the portrait image group to obtain the adjusted portrait image group; and fusing the adjusted portrait image and the entire background image to obtain the portrait video frame comprises: fusing the adjusted portrait image group and the entire background image to obtain the portrait video frame; wherein after the adjusted portrait image group is obtained, the method further comprises: determining a rotation percentage for each pixel in the adjusted portrait image group based on a distance of the pixel from a center point of the video frame; determining a rotation parameter of the pixel based on the rotation percentage and the set rotation angle, wherein the set rotation angle is related to a moment of the video frame in the video to be processed; and rotating the pixel according to the rotation parameter; wherein determining the rotation parameter of the pixel based on the rotation percentage and the set rotation angle comprises: determining an intermediate rotation angle based on the rotation percentage and the set rotation angle; and using a sine value of the intermediate rotation angle as a first sub-rotation parameter, and a cosine value of the intermediate rotation angle as a second sub-rotation parameter; and rotating the pixel according to the rotation parameter comprises: determining coordinate information of the rotated pixel based on the first sub-rotation parameter and the second sub-rotation parameter.

2 . The method according to claim 1 , wherein performing portrait segmentation on the video frame to obtain the portrait image and the background image comprises:
performing portrait recognition on each video frame to obtain a portrait mask image and a background mask image; obtaining the portrait image based on the portrait mask image and the video frame; and obtaining the background image based on the background mask image and the video frame.

3 . The method according to claim 1 , wherein after the adjusted portrait image group is obtained, the method further comprises: scaling at least one portrait image in the adjusted portrait image group by a set ratio to obtain the scaled portrait image group.

4 . The method according to claim 1 , wherein processing the background image to obtain the entire background image comprises:
obtaining optical flow information of a background image of a first video frame of the video to be processed or a background image of a previous video frame of the video frame; and processing the optical flow information using a set inpainting algorithm, to obtain the entire background image of the video frame.

5 . The method according to claim 1 , wherein the defined condition is that a distance of pixels from a center point of the video frame is greater than or less than a set value; and the set value is related to a moment of the video frame in the video to be processed.

6 . An electronic device, comprising: at least one processing apparatus; and a storage apparatus configured to store at least one program, wherein the at least one program, when executed by the at least one processing apparatus, causes the at least one processing apparatus to: obtain a video frame included in a video to be processed, wherein the video to be processed includes a portrait; perform portrait segmentation on the video frame to obtain a portrait image and a background image; adjust transparency of pixels that meet a defined condition in the portrait image to obtain the adjusted portrait image; process the background image to obtain an entire background image; fuse the adjusted portrait image and the entire background image to obtain a portrait video frame; and stitch a plurality of portrait video frames to obtain a target video;
wherein adjusting the transparency of the pixels that meet the defined condition in the portrait image to obtain the adjusted portrait image by: making at least one copy of the portrait image to obtain at least one portrait image copy; rotating the at least one portrait image copy about a coordinate axis of a three-dimensional space by a set angle to obtain the rotated portrait image copy, wherein the portrait image and the at least one rotated portrait image copy form a portrait image group; and adjusting transparency of pixels that meet the defined condition in the portrait image group to obtain the adjusted portrait image group; and fusing the adjusted portrait image and the entire background image to obtain the portrait video frame comprises: fusing the adjusted portrait image group and the entire background image to obtain the portrait video frame; wherein after the adjusted portrait image group is obtained, the device is further caused to: determine a rotation percentage for each pixel in the adjusted portrait image group based on a distance of the pixel from a center point of the video frame; determine a rotation parameter of the pixel based on the rotation percentage and the set rotation angle, wherein the set rotation angle is related to a moment of the video frame in the video to be processed; and rotate the pixel according to the rotation parameter; wherein determining the rotation parameter of the pixel based on the rotation percentage and the set rotation angle by: determining an intermediate rotation angle based on the rotation percentage and the set rotation angle; and using a sine value of the intermediate rotation angle as a first sub-rotation parameter, and a cosine value of the intermediate rotation angle as a second sub-rotation parameter; and wherein the device is caused to rotate the pixel according to the rotation parameter by determining coordinate information of the rotated pixel based on the first sub-rotation parameter and the second sub-rotation parameter.

7 . The device according to claim 6 , wherein the device is further caused to perform portrait segmentation on the video frame to obtain the portrait image and the background image by:
perform portrait recognition on each video frame to obtain a portrait mask image and a background mask image; obtain the portrait image based on the portrait mask image and the video frame; and obtain the background image based on the background mask image and the video frame.

8 . The device according to claim 6 , wherein after the adjusted portrait image group is obtained, the device is further caused to: scale at least one portrait image in the adjusted portrait image group by a set ratio to obtain the scaled portrait image group.

9 . The device according to claim 6 , wherein the device is further caused to pr

Assignees

• Beijing Bytedance Network Tech Co Ltd

Inventors

• Chen Yixin

Classifications

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T3/60

  Rotation of whole images or parts thereof · CPC title

• G06T3/40 Primary

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• G06T7/194

  involving foreground-background segmentation · CPC title