System for video super resolution using semantic components

Patent metadata

Field	Value
Publication number	US-2016253784-A1
Application number	US-201615154058-A
Country	US
Kind code	A1
Filing date	May 13, 2016
Priority date	Dec 11, 2014
Publication date	Sep 1, 2016
Grant date	—

Abstract

A method for increasing the resolution of a series of low resolution frames of a low resolution video sequence to a series of high resolution frames of a high resolution video sequence includes receiving the series of low resolution frames of the video sequence. The system determines a first plurality of semantically relevant key points of a first low resolution frame of the series of low resolution frames and determines a second plurality of semantically relevant key points of a second low resolution frame of the series of low resolution frames. The system temporally processes the first plurality of key points based upon the second plurality of key points to determine a more temporally consistent set of key points for the first plurality of key points.

First claim

I/We claim:

1 . A method for modifying the resolution of a series of frames of a video sequence using an image processing system comprising:
(a) receiving said series of frames of said video sequence by said image processing system; (b) determining a first plurality of key points of a first frame of said series of frames based upon semantic content of said first frame of said series of frames by said image processing system; (c) based on said first plurality of key points, determining a first set of candidate images in a database that match a semantic region of said first frame; (d) determining a second plurality of key points of a second frame of said series of frames based upon semantic content of said second frame of said series of frames by said image processing system; (e) based on said second plurality of key points, determining a second set of candidate images in said database that match a semantic region of said second frame; (f) selecting a selected candidate image in said database, based on said first set of candidate images and said second set of candidate images; (g) modifying said resolution of said second frame of said series of frames based on said selected candidate image in said database.

2 . The method of claim 1 wherein said determining said first set of candidate images in said database is based upon a score.

3 . The method of claim 2 wherein score is based upon pixel values.

4 . The method of claim 3 wherein said pixel values includes at least one of difference pixel values and pixel gradient values.

5 . The method of claim 1 wherein said selecting said selected candidate image is based upon temporal consistency.

6 . The method of claim 5 wherein said temporal consistency is based upon a frequency of semantic content over a plurality of said series of said frames.

7 . The method of claim 5 wherein said temporal consistency is based upon a matching strength of semantic content over a plurality of said series of said frames.

8 . The method of claim 1 wherein said first set of candidate images in said database are selected as a first set of best matching candidate images.

9 . The method of claim 1 wherein a plurality of said set of candidate images in said database are ranked over a temporal time period.

10 . The method of claim 9 wherein said ranking is based upon a histogram.

11 . The method of claim 1 wherein a plurality of said set of candidate images is selectively modified based upon individual frames of said series of frames and temporal consistency over a plurality of said frames of said series of frames.

12 . The method of claim 1 further comprising said modifying said resolution of said second frame of said series of frames based upon a selected candidate image from said database.

13 . The method of claim 12 further comprising said modifying based upon said selected candidate image is based upon a blending process.

Assignees

• Sharp Laboratories America Inc

Inventors

• Chen Xu
• Choudhury Anustup Kumar A
• Segall Christopher A
• Van Beek Petrus J L

Classifications

• G06T3/4007

  based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T2207/30201

  Face · CPC title

• G06T7/204

  Physics · mapped topic

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title