Frame rate conversion based on object tracking

Patent metadata

Field	Value
Publication number	US-9621841-B1
Application number	US-201514986554-A
Country	US
Kind code	B1
Filing date	Dec 31, 2015
Priority date	Dec 31, 2015
Publication date	Apr 11, 2017
Grant date	Apr 11, 2017

Abstract

A method, apparatus, and computer readable medium for frame rate conversion based on object tracking. The method includes determining whether a block of a first frame of an input video stream corresponds to a detected object region or a non-object region. The method further includes selecting a motion estimation operation from a group of operations comprising an object tracking operation based at least on the determining of whether the block of the first frame corresponds to the detected object region or the non-object region. The method further includes determining a motion vector of the block by performing the selected motion estimation operation and determining an interpolated frame based at least on the determined motion vector and the first frame and a second frame of the input video stream.

First claim

What is claimed is:

1. A method for frame rate conversion, the method comprising:
determining, by at least one processor, whether a block of a first frame of an input video stream corresponds to a detected object region or a non-object region;
based at least on the determining of whether the block of the first frame corresponds to the detected object region or the non-object region, selecting a motion estimation operation from a group of operations comprising an object tracking operation;
determining a motion vector of the block by performing the selected motion estimation operation; and
determining an interpolated frame based at least on the determined motion vector and the first frame and a second frame of the input video stream.

2. The method of claim 1 , wherein:
the group of operations further comprises a block matching operation, and
the selecting is in accordance with selecting the object tracking operation based at least on a determination that the block corresponds to the detected object region and selecting the block matching operation based at least on a determination that the block corresponds to the non-object region.

3. The method of claim 1 , wherein the determining of whether the block of the first frame of the input video stream corresponds to the detected object region or the non-object region comprises:
identifying a boundary of the detected object region in each of the first frame and the second frame of the input video stream; and
identifying a boundary of the block in each of the first frame and the second frame of the input video stream using a block mask.

4. The method of claim 1 , wherein the determining of the motion vector of the block comprises:
generating the motion vector based on a center position of each block of the first frame and the second frame of the input video stream corresponding to the detected object region or the non-object region; and
searching the block in accordance with the motion vector to generate an estimated motion vector.

5. The method of claim 4 , wherein the motion vector is a block-level motion vector.

6. The method of claim 4 , wherein the estimated motion vector is a recursive motion vector.

7. The method of claim 6 , wherein the recursive motion vector represents a pixel-level motion of the detected object region.

8. The method of claim 6 , wherein the recursive motion vector represents a pixel-level motion of the non-object region.

9. The method of claim 1 , further comprising:
displaying the interpolated frame on a display,
wherein the display is located in one of a television, a monitor, a laptop computer, a tablet computer, and a mobile phone.

10. An apparatus to convert frame rate, the apparatus comprising:
a memory; and
at least one processor operably connected to the memory, the at least one processor configured to:
determine whether a block of a first frame of an input video stream corresponds to a detected object region or a non-object region;
based at least on the determining of whether the block of the first frame corresponds to the detected object region or the non-object region, select a motion estimation operation from a group of operations comprising an object tracking operation;
determine a motion vector of the block by performing the selected motion estimation operation; and
determine an interpolated frame based at least on the determined motion vector and the first frame and a second frame of the input video stream.

11. The apparatus of claim 10 , wherein:
the group of operations further comprises a block matching operation, and
the selection is in accordance with selecting the object tracking operation based at least on a determination that the block corresponds to the detected object region and the selection of the block matching operation based at least on a determination that the block corresponds to the non-object region.

12. The apparatus of claim 10 , wherein to determine whether the block of the first frame of the input video stream corresponds to the detected object region or a non-object region, the at least one processor configured is further configured to:
identify a boundary of the detected object region each of the first frame and the second frame of the input video stream; and
identify a boundary of the block each of the first frame and the second frame of the input video stream using a block mask.

13. The apparatus of claim 10 , wherein to determine the motion vector of the block, the at least one processor configured is further configured to:
generate the motion vector based on a center position of each block of the first frame and the second frame of the input video stream corresponding to the detected object region or the non-object region; and
search of the block in accordance with the motion vector to generate an estimated motion vector.

14. The apparatus of claim 13 , wherein the motion vector is a block-level motion vector.

15. The apparatus of claim 13 , wherein the estimated motion vector is a recursive motion vector.

16. The apparatus of claim 13 , wherein the recursive motion vector represents a pixel-level motion of the detected object region.

17. The apparatus of claim 10 , further comprising:
a display configured to display the interpolated frame,
wherein the apparatus is one of a television, a monitor, a laptop computer, a tablet computer, and a mobile phone.

18. A non-transitory computer readable medium comprising program code, that when executed by at least one processor, causes the at least one processor to:
determine whether a block of a first frame of an input video stream corresponds to a detected object region or a non-object region;
based at least on the determining of whether the block of the first frame corresponds to the detected object region or the non-object region, select a motion estimation operation from a group of operations comprising an object tracking operation;
determine a motion vector of the block by performing the selected motion estimation operation; and
determine an interpolated frame based at least on the determined motion vector and the first frame and a second frame of the input video stream.

19. The non-transitory computer readable medium of the claim 18 , wherein the program code, that when executed by the at least one processor, causes the at least one processor to determine whether the block of the first frame of the input video stream corresponds to the detected object region or the non-object region comprises program code, that when executed by at least one processor, causes the at least one processor to:
identify a boundary of the detected object region each of the first frame and the second frame of the input video stream; and
identify a boundary of the block each of the first frame and the second frame of the input video stream using a block mask.

20. The non-transitory computer readable medium of the claim 18 , wherein the program code, that when executed by the at least one processor, causes the at least one processor to determine the motion vector of the block comprises program code, that when executed by at least one processor, causes the at least one processor to:
generate the motion vector based on a center position of each block of the first frame and the second frame of the input video stream corresponds to the detected object region or the non-object region; and
search of the block in accordance with the motion vector to generate an estimated motion vector.

Assignees

• Samsung Electronics Co Ltd
• Samsung Electronics Co Ltd

Inventors

• Hong Hunsop
• Kim Yeong-Taeg

Classifications

• H04N23/951

  by using two or more images to influence resolution, frame rate or aspect ratio · CPC title

• G09G2340/0435

  Change or adaptation of the frame rate of the video stream · CPC title

• H04N19/577 Primary

  Motion compensation with bidirectional frame interpolation, i.e. using B-pictures · CPC title

• H04N7/0127 Primary

  by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter · CPC title

• H04N5/145

  Movement estimation (for video coding H04N19/51) · CPC title