Methodology and apparatus for generating high fidelity zoom for mobile video

Patent metadata

Field	Value
Publication number	US-10097765-B2
Application number	US-201615134294-A
Country	US
Kind code	B2
Filing date	Apr 20, 2016
Priority date	Apr 20, 2016
Publication date	Oct 9, 2018
Grant date	Oct 9, 2018

Abstract

The method includes in order to generate a composite image: identifying, in the frame of a video stream captured by a camera, a motion characteristic associated with moving objects in a scene while the camera captured a sliding window of the video stream. The method includes for a plurality of frames in the sliding window: controlling, by the processing circuitry, a weight of blending of the frame based on the identified motion characteristic to enable the composite image to be generated according to the controlled weights of blending of the plurality of frames in the sliding window.

First claim

What is claimed is:

1. An apparatus comprising:
a camera configured to capture a video stream at a capture frame rate; and
a processing circuitry configured to, in order to generate a composite image:
identify, in a frame of the video stream captured by the camera, a motion characteristic associated with moving objects in a scene while the camera captured a sliding window of the video stream; and
for a plurality of frames in the sliding window, control a weight of blending of the frame based on the identified motion characteristic to enable the composite image to be generated according to the controlled weights of blending of the plurality of frames in the sliding window;

wherein the processing circuitry is further configured to, in order to generate the composite image having a user-selected digital zoom ratio:
include at least two frames of the video stream in the sliding window that contains a number N of consecutive frames of the video stream,
determine camera motion information associated with motion of the camera while the camera captured the video stream, and
for each frame in one or more frames of the video stream:
analyze an amount of warp of the frame based on the camera motion information, and
generate a set of warp parameters indicating the amount of warp of the frame;
for frames in the sliding window that includes the frame:
generate a warp interpolated frame by applying the warp parameters and an image interpolation technique to the frame,

wherein the processing circuitry is further configured to:
generate the composite image by: blending the warp interpolated frames corresponding to the sliding window into a single-blended frame according to the controlled weight of blending of each frame in the sliding window, and
produce an output video stream including the composite image generated from the sliding window and a subsequent composite image generated from a subsequent sliding window that includes a subsequent N consecutive frames of the video stream.

2. The apparatus of claim 1 , wherein the processing circuitry is further configured to:
determine a user-selected zoom ratio, wherein the user-selected zoom ratio equals a first partial zoom ratio multiplied by a second partial zoom ratio;
prior to generating the warp interpolated frame, upscale each frame in the sliding window by the second partial zoom ratio; and
generate the warp interpolated frame by: upscaling each frame in the sliding window by the first partial zoom ratio by applying a multi-frame interpolation technique, such that the composite image is scaled to the user-selected zoom ratio.

3. The apparatus of claim 1 , wherein the processing circuitry is configured to:
pre-process each frame of the video stream according to the capture frame rate by at least one of: enhancing sharpness or filtering noise of the frame;
control the pre-processing of each frame of the video stream by controlling a sharpness strength based on auto-focus information;
generate the composite image by: applying photometric (PM) alignment and blend techniques to the sliding window according to a rate N times the capture frame rate to match brightness and color to that of a reference frame in the sliding window; and
apply feature detection and feature tracking to the video stream to pre-processed frames output from the pre-processing operation.

4. The apparatus of claim 1 , wherein the processing circuitry is further configured to:
generate the composite image by: applying a photometric (PM) alignment technique to each frame of the sliding window according to the capture frame rate by adjusting brightness and color of the N frames of the sliding window to a time-varying target.

5. The apparatus of claim 1 , wherein the processing circuitry is further configured to: blend the warp interpolated frames corresponding to the sliding window by:
generating local blend statistics including at least one of:
a blending weight, or
a mean motion; and

perform the blending according to a statistical function including at least one of: a mean or a median.

6. The apparatus of claim 1 , wherein, to reduce end-to-end latency, the processing circuitry is configured to:
select a reference frame from the sliding window by one of:
selecting a most recent captured frame as the reference frame, or
selecting a center frame from among the N frames as the reference to reduce motion distortion;

generate the composite image by:
post-processing of the single-blended frame, and
controlling a strength of the post-processing by using local motion statistics to control a strength of at least one of:
post-processing noise control,
post-processing sharpness control, or
post-processing contrast control; and

use the output video stream at least one of:
an input of an image signal processor (ISP) of the camera,
an intermediate point in the ISP with intermediate data format, or
execution of a video playback application.

7. The apparatus of claim 1 wherein the processing circuitry is configured to:
control the camera by modifying an image signal processor (ISP) of the camera for at least one of: noise filtering, demosaicing configuration, sharpness enhancement, defect pixel correction, color or exposure;
control the weight of blending of each frame based on camera International Organization of Standardization (ISO) parameters;
operate in High Fidelity (HiFi) video mode based on at least one of:
an automatically selection of the HiFi video mode in response to the user-selected digital zoom ratio exceeding a threshold digital zoom ratio,
an automatically selection of the HiFi video mode based on a pre-selected quality level,
receiving a user input indicating a selection of the HiFi video mode;

determine the camera motion information by using at least one of:
using digital analysis,
inertial measurement received from an inertial measurement unit (IMU), or
video stabilization information from the camera;

identify a presence of Electronic Rolling Shutter (ERS) distortion and determine parameters of the ERS distortion based on the video stream and the camera motion information; and
dynamically select the number N of consecutive frames in each sliding window based on at least one of:
a pre-determined calculation of apparatus hardware capabilities,
camera International Organization of Standardization (ISO) parameters, or
scene brightness.

8. The apparatus of claim 1 , wherein the processing circuitry is further configured to:
decompose the camera motion information into intended camera motion path and hand-shake camera motion based on the camera motion information,
wherein to make the intended camera motion path aesthetically pleasing, generate a camera stabilization path to be a function of the user intended camera motion path by removing sudden jerks; and
generate the set of warp parameters by: combining warp correction for camera motion with at least one of:
warp correction for camera motion stabilization, or
warp correction for Electronic Rolling Shutter correction.

9. The apparatus of claim 1 , wherein the processing circuitry is configured to:
determine to accept or reject each frame in the sliding window based on suitability for blending with other frames in the sliding window;
determine a global frame weight control for each frame in the video stream based on at least one of:
a sharpness of the frame,
an estimate of amount of motion in the frame,
an estimate of amount of noise in the frame,
a difference of time between the frame and a reference frame in the sliding window, or
an estimate of amount of details in frame;

adjust the weight of blending of the frame by rejecting each frame in the sliding window tha

Assignees

• Samsung Electronics Co Ltd
• Samsung Electronics Co Ltd

Inventors

• Sheikh Hamid Rahim
• Pekkucuksen Ibrahim
• Glotzbach John

Classifications

• H04N23/69 Primary

  Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming · CPC title

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title

• H04N23/6811 Primary

  based on the image signal · CPC title

• H04N5/144

  Movement detection (for video coding H04N19/503; analysis of motion in general G06T7/20) · CPC title

• H04N23/689

  Motion occurring during a rolling shutter mode · CPC title