System and method for multiple-frame based super resolution interpolation for digital cameras

Patent metadata

Field	Value
Publication number	US-9282253-B2
Application number	US-201414183082-A
Country	US
Kind code	B2
Filing date	Feb 18, 2014
Priority date	Feb 18, 2014
Publication date	Mar 8, 2016
Grant date	Mar 8, 2016

Abstract

A digital camera system for super resolution image processing constructed to receive a plurality of input frames and output at least one digitally zoomed frame is provided. The digital camera system includes a motion registration module configured to generate motion information associated with the plurality of input frames, an interpolation module configured to generate a plurality of interpolated input frames based at least in part on the plurality of input frames and the motion information, a weights calculation module configured to calculate one or more weights associated with the plurality of input frames based on at least the motion information, and a weighted merging module configured to merge the plurality interpolated input frames consistent with the one or more weights to generate the at least one digitally zoomed frame.

First claim

What is claimed is:

1. A digital camera system for super resolution image processing constructed to receive a plurality of input frames and output at least one digitally zoomed frame, the digital camera system comprising at least one processor configured to:
generate motion information associated with the plurality of input frames;
generate a plurality of interpolated input frames based at least in part on the plurality of input frames and the motion information;
calculate one or more weights associated with the plurality of input frames based on at least the motion information; and
merge the plurality of interpolated input frames consistent with the one or more weights to generate the at least one digitally zoomed frame and generate a frame weight map for the digitally zoomed frame, the frame weight map including weight information based on the one or more weights.

2. The digital camera system of claim 1 , wherein the at least one processor is further configured to calculate a weighted median of the interpolated input frames based on the one or more weights.

3. The digital camera system of claim 1 , wherein the at least one processor is further configured to compute at least one of the one or more frame weights based on a Bayer-pattern configuration of a plurality of photoreceptors in an image sensor of the digital camera system.

4. The digital camera system of claim 3 , wherein the at least one processor is further configured to compute at least one of the one or more weights based on an estimated accuracy level of the motion information.

5. The digital camera system of claim 4 , wherein the at least one processor is further configured to compute at least one of the one or more frame weights based on a level of detail in at least one of the plurality of frames.

6. The digital camera system of claim 1 , wherein the at least one processor is further configured to compute at least one of the one or more weights based on an estimated accuracy level of the motion information.

7. The digital camera system of claim 6 , wherein the at least one processor is further configured to compute at least one of the one or more frame weights based on a level of detail in at least one of the plurality of frames.

8. The digital camera system of claim 1 , wherein the plurality of input frames includes at least a current input frame and one of a previous input frame and a previous output frame.

9. The digital camera system of claim 1 , wherein the at least one processor is further configured to compute at least one of the one or more weights based on the frame weight map of a previous output frame.

10. The digital camera system of claim 1 , wherein the plurality of input frames have a first frame rate and the at least one digitally zoomed frame has a second frame rate that is less than the first rate.

11. A method of super resolution image processing in a digital camera, the method comprising:
receiving a plurality of input frames;
generating motion information associated with the plurality of input frames;
interpolating the plurality of input frames based on the plurality of input frames and
the motion information to generate a plurality of interpolated input frames;
calculating one or more weights associated with the plurality of input frames based on at least the motion information; and
merging the plurality of interpolated input frames consistent with the one or more weights associated with the plurality of frames to generate at least one digitally zoomed frame and generating a frame weight map for the digitally zoomed frame, the frame weight map including weight information based on the one or more weights.

12. The method of claim 11 , wherein merging the plurality of interpolated input frames includes calculating a weighted median of the interpolated input frames based on the one or more weights.

13. The method of claim 11 , wherein calculating the one or more weights includes calculating at least one of the one or more weights based on a Bayer-pattern configuration of a plurality of photoreceptors in an image sensor of the digital camera.

14. The method of claim 13 , wherein calculating the one or more weights further includes calculating at least one of the one or more weights based on an estimated accuracy level of the motion information.

15. The method of claim 14 , wherein calculating the one or more weights further includes calculating at least one of the one or more weights based on a level of detail in at least one of the plurality of input frames.

16. The method of claim 11 , wherein receiving the plurality of input frames includes receiving at least a current input frame and one of a previous input frame and a previous output frame.

17. The method of claim 11 , wherein calculating the one or more weights includes calculating at least one of the one or more weights based on the frame weight map of a previous frame.

18. A non-transitory computer readable medium having stored thereon sequences of instructions for super resolution image processing that instruct at least one processor to:
receive a plurality of input frames;
generate motion information associated with the plurality of input frames;
interpolate the plurality of input frames based on the plurality of input frames and the motion information to generate a plurality of interpolated input frames;
calculate one or more weights associated with the plurality of input frames based on at least the motion information and a level of detail in at least one of the plurality of frames; and
merge the plurality of interpolated input frames consistent with the one or more weights associated with the plurality of frames to generate at least one digitally zoomed frame and generate a frame weight map for the digitally zoomed frame, the frame weight map including weight information based on the one or more weights.

Assignees

• Qualcomm Technologies Inc

Inventors

• Amitay Amichay
• Levy Noam

Classifications

• H04N25/48

  Increasing resolution by shifting the sensor relative to the scene · CPC title

• H04N23/951

  by using two or more images to influence resolution, frame rate or aspect ratio · CPC title

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• H04N1/3935

  with modification of image resolution, i.e. determining the values of picture elements at new relative positions · CPC title

• H04N23/69 Primary

  Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming · CPC title