Methods and systems for generating enhanced images using multi-frame processing

Patent metadata

Field	Value
Publication number	US-2016267349-A1
Application number	US-201514715561-A
Country	US
Kind code	A1
Filing date	May 18, 2015
Priority date	Mar 11, 2015
Publication date	Sep 15, 2016
Grant date	—

Abstract

Examples of the disclosure enable multi-frame processing of images to be efficiently performed. In some examples, one or more interest points are identified in a plurality of images. One or more features are extracted from the one or more interest points using an extraction algorithm. Based on the one or more extracted features, the plurality of images are registered to generate a plurality of registered images. The registered plurality of images are combined to generate a composite image. Aspects of the disclosure facilitate increasing speed, conserving memory, reducing processor load or an amount of energy consumed, and/or reducing network bandwidth usage.

First claim

What is claimed is:

1 . A computer-implemented method for processing a multi-frame image, the method comprising executing on a computing device the operations of:
identifying one or more interest points in a plurality of images; extracting, using the computing device, one or more features from the interest points using one or more of a filter module, a gradient module, a pooler module, and a normalizer module; based on the extracted features, registering the plurality of images using one or more of a homography estimation module and a warp module to generate a plurality of registered images; and combining the plurality of registered images to generate a composite image for one or more of high-dynamic range imaging, super high-dynamic range imaging, de-noising, image stabilizing, de-blurring, super-resolution imaging, de-hazing, panoramic stitching, depth of field stacking, and rolling shutter correcting.

2 . The computer-implemented method of claim 1 , wherein identifying one or more interest points comprises detecting one or more corners, arches, edges, blobs, ridges, textures, colors, differentials, or lighting changes in the plurality of images, wherein a first corner, arch, edge, blob, ridge, texture, color, differential, or lighting change corresponds to a first interest point.

3 . The computer-implemented method of claim 1 , wherein extracting one or more features comprises smoothing, by the filter module, one or more pixels associated with the interest points.

4 . The computer-implemented method of claim 1 , wherein extracting one or more features comprises:
computing, by the gradient module, one or more gradients along a first axis and a second axis perpendicular to the first axis; and based on the computed gradients, generating, by the gradient module, an output array including one or more feature maps.

5 . The computer-implemented method of claim 1 , wherein extracting one or more features comprises pooling, by the pooler module, one or more feature maps along a grid, wherein the feature maps correspond to the extracted features.

6 . The computer-implemented method of claim 1 , wherein registering the plurality of images comprises registering the plurality of images with respect to a common coordinate system.

7 . The computer-implemented method of claim 1 , wherein registering the plurality of images comprises warping one or more images of the plurality of images using one or more affine transforms.

8 . The computer-implemented method of claim 1 , further comprising:
retrieving the plurality of images from a first bus; and transmitting the plurality of registered images to a second bus different from the first bus.

9 . A mobile device comprising:
a sensor module configured to capture data corresponding to a plurality of images; a memory area storing computer-executable instructions for processing a multi-frame image based on the plurality of images; and a processor configured to execute the computer-executable instructions to:
identify one or more interest points in the plurality of images;
extract one or more features from the interest points;
based on the extracted features, register the plurality of images to generate a plurality of registered images; and
combine the plurality of registered images to generate a composite image for one or more of high-dynamic range imaging, super high-dynamic range imaging, de-noising, image stabilizing, de-blurring, super-resolution imaging, de-hazing, panoramic stitching, depth of field stacking, and rolling shutter correcting.

10 . The mobile device of claim 9 , wherein the processor is configured to execute the computer-executable instructions to detect one or more corners, arches, edges, blobs, ridges, textures, colors, differentials, or lighting changes in the plurality of images, wherein a first corner, arch, edge, blob, ridge, texture, color, differential, or lighting change corresponds to a first interest point.

11 . The mobile device of claim 9 , wherein the processor is configured to execute the computer-executable instructions to smooth one or more pixels associated with the interest points.

12 . The mobile device of claim 9 , wherein the processor is configured to execute the computer-executable instructions to:
compute one or more gradients along a first axis and a second axis perpendicular to the first axis; and based on the one or more computed gradients, generate an output array including one or more feature maps.

13 . The mobile device of claim 9 , wherein the processor is configured to execute the computer-executable instructions to pool one or more feature maps along a grid, wherein the feature maps correspond to the extracted features.

14 . The mobile device of claim 9 , wherein the processor is configured to execute the computer-executable instructions to register the plurality of images with respect to a common coordinate system.

15 . The mobile device of claim 9 , wherein the processor is configured to execute the computer-executable instructions to warp one or more images of the plurality of images.

16 . The mobile device of claim 9 , further comprising a plurality of busses, wherein the sensor module is configured to transmit the plurality of images to a first bus of the plurality of busses, and the processor is configured to execute the computer-executable instructions to:
retrieve the plurality of images from the first bus; transmit the plurality of registered images to a second bus of the plurality of busses.

17 . A system comprising:
a sensor module configured to capture data corresponding to a plurality of images, and transmit the plurality of images to one or more of a first frame bus and a first network location; an image sensor processor module configured to retrieve the plurality of images from one or more of the first frame bus and the first network location, process the plurality of images, and transmit the plurality of processed images to one or more of the first frame bus and the first network location; an accelerator module configured to retrieve the plurality of processed images from one or more of the first frame bus and the first network location, register the plurality of processed images, and transmit the plurality of registered images to one or more of a second frame bus and a second network location; and a processor module configured to retrieve the plurality of registered images from one or more of the second frame bus and the second network location, and combine the plurality of registered images to generate a composite image for one or more of high-dynamic range imaging, super high-dynamic range imaging, de-noising, image stabilizing, de-blurring, super-resolution imaging, de-hazing, panoramic stitching, depth of field stacking, and rolling shutter correcting.

18 . The system of claim 17 , wherein the accelerator module is configured to:
identify one or more interest points in the plurality of images; extract one or more features from the interest points; and based on the extracted features, register the plurality of images with respect to a common coordinate system to generate the plurality of registered images.

19 . The system of claim 17 , wherein at least the sensor module and the accelerator module are at a mobile device.

20 . The system of claim 17 , wherein at least the sensor module is at a mobile device, and at least the accelerator module is at a server coupled to the mobile device.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Shoaib Mohammed
• Liu Jie
• Stoakley Richard Wales
• Uyttendaele Matthieu Tony

Classifications

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• G06F17/3028

  Physics · mapped topic

• G06T7/0024

  Physics · mapped topic

• G06T11/60

  Creating or editing images; Combining images with text · CPC title

• G06T5/00

  Image enhancement or restoration · CPC title