Camera parameter-assisted video frame rate up conversion

Patent metadata

Field	Value
Publication number	US-9609331-B2
Application number	US-201514832944-A
Country	US
Kind code	B2
Filing date	Aug 21, 2015
Priority date	May 26, 2010
Publication date	Mar 28, 2017
Grant date	Mar 28, 2017

Abstract

This disclosure describes methods and apparatus for decoding data. In one aspect, the method comprises decoding encoded video data to obtain decoded video frame data, the encoded video data comprising encoded video frame data encoded at a first frame rate and embedded data. The method further comprises determining a camera parameter from the embedded data and up-converting the decoded video frame data to a second frame rate based on the camera parameter. The determined camera parameter may be, for example, a parameter associated with one or more of a zoom factor, an auto focus status, lens position information, frame luma information, an auto exposure (AE) convergence status, an automatic white balance (AWB) convergence status, global motion information, and frame blurriness information, and the like. An encoding device may embed the camera parameter(s) in an encoded video bit stream for a decoder to utilize during frame rate up-conversion.

First claim

What is claimed is:

1. A method for decoding data comprising:
decoding encoded video data to obtain decoded video frame data, the encoded video data comprising encoded video frame data and embedded data, the encoded video frame data encoded at a first frame rate;
determining a camera parameter from the embedded data; and
up-converting the decoded video frame data to a second frame rate based on the camera parameter.

2. The method of claim 1 , wherein the encoded video frame data further comprises a plurality of video frames, and wherein the embedded data comprises a plurality of camera parameters corresponding to the plurality of video frames.

3. The method of claim 2 , further comprising:
determining zooming information from at least one of the plurality of camera parameters,
wherein up-converting the decoded video frame data to the second frame rate comprises:
determining a zooming factor from the zooming information,
scaling one or more of the plurality of video frames based on the zooming factor, and
generating one or more additional video frames to up-convert the decoded video frame data by interpolating one or more of the plurality of video frames based at least in part on the scaled one or more of the plurality of video frames.

4. The method of claim 3 , further comprising:
determining a motion vector based at least in part on the scaled one or more of the plurality of video frames,
wherein the interpolating of the one or more of the plurality of video frames is further based at least in part on the motion vector.

5. The method of claim 2 , further comprising:
determining auto-focus information and blurriness information from at least one of the plurality of camera parameters,
wherein up-converting the decoded video frame data to the second frame rate comprises:
determining, based at least in part on the auto-focus information, that at least one of the plurality of video frames was captured during focusing,
determining, in response to the least one of the plurality of video frames being captured during focusing, whether an average blurriness level of the plurality of video frames exceeds a threshold, based on the blurriness information, and
generating, in response to the average blurriness level of the plurality of video frames exceeding the threshold, one or more additional video frames to up-convert the decoded video frame data by interpolating one or more of the plurality of video frames based at least in part on an average of pixel values of the plurality of video frames.

6. The method of claim 5 , wherein up-converting the decoded video frame data to the second frame rate comprises:
determining, in response to the average blurriness level of the plurality of video frames not exceeding the threshold, a length of a motion vector search range to be inversely proportional to a blurriness level of the plurality of video frames, and
generating one or more additional video frames to up-convert the decoded video frame data by interpolating the plurality of video frames based on the determined motion vector search range.

7. The method of claim 2 , further comprising:
determining brightness information from at least one of the plurality of camera parameters,
wherein up-converting the decoded video frame data to the second frame rate comprises:
determining, based on the brightness information, a frame luma of a first video frame and a second video frame from the plurality of video frames, respectively,
generating a first reference frame based on the first video frame and the frame luma of the first video frame,
generating a second reference frame based on the second video frame and the frame luma of the first video frame,
determining a motion vector based at least in part on the first and second reference frame, and
generating one or more additional video frames to up-convert the decoded video frame data by interpolating one or more of the plurality of video frames based at least in part on the determined motion vector and the first and second reference frames.

8. The method of claim 7 , further comprising adjusting a frame luma of the one or more additional video frames based at least in part on the frame luma of the first and second video frames, wherein the adjusted frame luma of the one or more additional video frames is between the frame luma of the first and second video frames.

9. The method of claim 2 , further comprising:
determining auto-white balance (AWB) convergence status information and color gain information from at least one of the plurality of camera parameters,
wherein up-converting the decoded video frame data to the second frame rate comprises:
generating one or more additional video frames to up-convert the decoded video frame data by interpolating one or more of the plurality of video frames, and
adjusting a color gain of the one or more additional video frames based on the AWB convergence status information and the color gain information.

10. The method of claim 2 , further comprising:
determining global motion information from at least one of the plurality of camera parameters,
wherein up-converting the decoded video frame data to the second frame rate comprises:
determining a motion vector search range based at least in part on the global motion information,
determining motion vectors based on the motion vector search range, and
generating one or more additional video frames to up-convert the decoded video frame data by interpolating one or more of the plurality of video frames based at least in part on the determined motion vector.

11. The method of claim 1 , further comprising displaying the up-converted decoded video frame data at the second frame rate on an electronic display.

12. The method of claim 1 , wherein the camera parameter used for up-converting the decoded video frame data to the second frame rate includes one or more of a zoom factor, an auto focus status, lens position information, frame luma information, an auto exposure (AE) convergence status, an automatic white balance (AWB) convergence status, global motion information, and frame blurriness information.

13. An apparatus for decoding data comprising:
a decoder configured to:
decode encoded video data to obtain decoded video frame data, the encoded video data comprising encoded video frame data and embedded data, the encoded video frame data encoded at a first frame rate; and
determine a camera parameter from the embedded data; and

a processor configured to up-convert the decoded video frame data to a second frame rate based on the camera parameter.

14. The apparatus of claim 13 , wherein the decoder is further configured to:
decode the encoded video data to obtain a plurality of video frames; and
determine a plurality of camera parameters corresponding to the plurality of video frames from the embedded data.

15. The apparatus of claim 14 , wherein the processor is further configured to:
determine zooming information from at least one of the plurality of camera parameters;
determine a zooming factor from the zooming information;
scale one or more of the plurality of video frames based on the zooming factor; and
generate one or more additional video frames to up-convert the decoded video frame data to the second frame rate by interpolating one or more of the plurality of video frames based at least in part on the scaled one or more of the plurality of video frames.

16. The apparatus of claim 15 , wherein the processor is further configured to:
determine a motion vector based at least in part on the scaled one or more of the plura

Assignees

• Qualcomm Inc

Inventors

• Liang Liang
• Hung Bob R
• Dane Gokce

Classifications

• H04N21/2353

  specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata · CPC title

• H04N5/765

  Interface circuits between an apparatus for recording and another apparatus · CPC title

• H04N19/139

  Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability · CPC title

• H04N19/513

  Processing of motion vectors · CPC title

• H04N19/46

  Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) · CPC title