Multi-lens video recording method and related device

Patent metadata

Field	Value
Publication number	US-12382172-B2
Application number	US-202318189319-A
Country	US
Kind code	B2
Filing date	Mar 24, 2023
Priority date	Sep 27, 2020
Publication date	Aug 5, 2025
Grant date	Aug 5, 2025

Abstract

A multi-lens video recording method and a main control device are disclosed. The method includes: performing collaborative photographing through a plurality of cameras, determining a target camera movement mode based on main subject information in images obtained through photographing and motion information of the cameras, and then fusing, based on the target camera movement mode, the images obtained through photographing by using the plurality of cameras, to implement online camera movement fusion on the plurality of video stream images.

First claim

What is claimed is:

1. A multi-lens video recording method, comprising:
obtaining a plurality of first images and a plurality of second images, wherein the plurality of first images and the plurality of second images are respectively obtained through photographing a same photographed object by using a first camera and a second camera, and at least one parameter is different between the first camera and the second camera;
obtaining motion information of the first camera in a process of obtaining the plurality of first images through photographing;
obtaining main subject information in each of the plurality of first images;
determining a target camera movement mode based on the motion information of the first camera and the main subject information in each of the plurality of first images; and
fusing the plurality of first images and the plurality of second images based on the target camera movement mode, to obtain the target video,
wherein the determining a target camera movement mode based on the motion information of the first camera and the main subject information in each of the plurality of first images comprises:
determining, based on the motion information of the first camera, whether the first camera is displaced in the process of obtaining the plurality of first images;
determining the target camera movement mode based on the main subject information in each of the plurality of the first images after determining that the first camera is not displaced; and
determining the target camera movement mode based on a motion manner of the first camera and the main subject information in each of the plurality of first images after determining that the first camera is displaced.

2. The method according to claim 1 , wherein
a field of view (FOV) of the first camera is greater than an FOV of the second camera; or
a focal length of the first camera is less than a focal length of the second camera.

3. The method according to claim 1 , wherein the determining the target camera movement mode based on the main subject information in each of the plurality of first images comprises:
determining the target camera movement mode as a push-in lens when the main subject information in each of the plurality of first images indicates that each of the plurality of first images comprises a main subject, and a first ratio of each of the plurality of first images is less than a first preset ratio, wherein the first ratio of each first image is a ratio of an area of a region of interest (ROI) in which the main subject in the first image is located to an area of the first image; or the first ratio of each first image is a ratio of a width of an ROI in which the main subject in the first image is located to a width of the first image; or the first ratio of each first image is a ratio of a height of an ROI in which the main subject in the first image is located to a height of the first image; and
determining the target camera movement mode as a pull-out lens when the main subject information in each of the plurality of first images indicates that a default ROI region in each first image does not comprise the main subject, and an image in the default ROI in each first image is a part of the first image.

4. The method according to claim 1 , wherein the determining the target camera movement mode based on a motion manner of the first camera and the main subject information in each of the plurality of first images comprises:
determining the target camera movement mode as a traveling lens when the motion manner of the first camera indicates that the first camera moves in a same direction when obtaining the plurality of first images, and the main subject information in each of the plurality of first images indicates that main subjects in the plurality of first images change;
determining the target camera movement mode as a panning lens when the motion manner of the first camera indicates that the first camera performs a reciprocating motion in two opposite directions when obtaining the plurality of first images through photographing, and the main subject information in each of the plurality of first images indicates that the main subjects in the plurality of first images change; and
determining the target camera movement mode as a follow lens when the motion manner of the first camera indicates that the first camera moves when obtaining the plurality of first images, and the main subject information in each of the plurality of first images indicates that the main subjects in the plurality of first images do not change.

5. The method according to claim 3 , wherein when the target camera movement mode is the push-in lens or the pull-out lens, the fusing the plurality of first images and the plurality of second images based on the target camera movement mode, to obtain the target video comprises:
obtaining a preset region in each of the plurality of first images, wherein the preset region in each of the plurality of first images comprises the main subject, and when the target camera movement mode is the push-in lens, sizes of the preset regions in the plurality of first images are gradually reduced in a time sequence; and when the target camera movement mode is the pull-out lens, the sizes of the preset regions in the plurality of first images gradually increase in the time sequence;
obtaining a plurality of first image pairs from the plurality of first images and the plurality of second images, wherein each of the plurality of first image pairs comprises a first image and a second image that have a same timestamp;
obtaining a plurality of first target images from the plurality of first image pairs, wherein the plurality of first target images is in a one-to-one correspondence with the plurality of first image pairs, and each of the plurality of first target images is an image that has a smallest viewing angle range in a corresponding first image pair and that comprises a preset region in the first image;
cutting each of the plurality of first target images to obtain a part that overlaps an image in a preset region in a first image in a second image pair to which the first target image belongs, to obtain a plurality of third images; and
processing each of the plurality of third images to obtain a plurality of processed third images, wherein a resolution of each of the plurality of processed third images is a preset resolution, and
the target video comprises the plurality of processed third images.

6. The method according to claim 4 , wherein when the target camera movement mode is the traveling lens, the panning lens, or the follow lens, the fusing the plurality of first images and the plurality of second images based on the target camera movement mode, to obtain the target video comprises:
performing subject detection and extraction on the plurality of first images to obtain an ROI in each of the plurality of first images, and cutting the ROI in each of the plurality of first images to obtain a plurality of fourth images, wherein each of the plurality of fourth images comprises a main subject; and the plurality of first images are in a one-to-one correspondence with the plurality of fourth images, and a timestamp of each of the plurality of fourth images is the same as a timestamp of a first image corresponding to the fourth image;
processing each of the plurality of fourth images to obtain a plurality of fifth images, wherein a resolution of each of the plurality of fifth images is a preset resolution; and the plurality of fourth images are in a one-to-one correspondence with the plurality of fifth images, and a timestamp of each of the plurality of fifth images is the same as a timestamp of a fourth image corresponding to the fifth image;
obtaining a plurality of second image pairs from the plurality of second

Assignees

• Huawei Tech Co Ltd

Inventors

• Teng Fei
• Chen Junkai
• Xiao Zhaolei
• Wei He
• Hu Changqi
• Zhao Zhijie

Classifications

• H04N23/95

  Computational photography systems, e.g. light-field imaging systems · CPC title

• H04N5/265

  Mixing · CPC title

• H04N23/681

  Motion detection · CPC title

• H04N23/667

  Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes · CPC title

• H04N23/61

  based on recognised objects · CPC title