Tile-based digital image correspondence

Patent metadata

Field	Value
Publication number	US-9860441-B1
Application number	US-201715647532-A
Country	US
Kind code	B1
Filing date	Jul 12, 2017
Priority date	May 23, 2015
Publication date	Jan 2, 2018
Grant date	Jan 2, 2018

Abstract

A computing device may obtain a first captured image of a scene and a second captured image of the scene. For a plurality of m×n pixel tiles of the first captured image, the computing device may determine respective distance matrixes. The distance matrixes may represent respective fit confidences between the m×n pixel tiles and pluralities of target p×q pixel tiles in the second captured image. The computing device may approximate the distance matrixes with respective bivariate surfaces. The computing device may upsample the bivariate surfaces to obtain respective offsets for pixels in the plurality of m×n pixel tiles. The respective offsets, when applied to pixels in the plurality of m×n pixel tiles, may cause parts of the first captured image to estimate locations in the second captured image.

First claim

What is claimed is:

1. A method comprising:
capturing, by an image sensor of a camera device, a first captured image of a scene and a second captured image of the scene;
for a plurality of m×n pixel tiles of the first captured image, determining, by a processor of the camera device, respective distance matrixes, wherein the distance matrixes represent respective fit confidences between the m×n pixel tiles and pluralities of target p×q pixel tiles in the second captured image;
approximating, by the processor, the distance matrixes with respective bivariate surfaces;
upsampling, by the processor, the bivariate surfaces to obtain respective offsets for pixels in the plurality of m×n pixel tiles, such that the respective offsets, when applied to pixels in the plurality of m×n pixel tiles, cause parts of the first captured image to estimate locations in the second captured image; and
based on the first captured image, the second captured image, and the respective offsets, generating, by the processor, an intermediate image with intermediate locations of the pixels from the first captured image and the second captured image, wherein the intermediate image represents an intermediate version of the scene that is temporally or physically between those of the first captured image and the second captured image.

2. The method of claim 1 , wherein upsampling the bivariate surfaces uses bicubic interpolation of respective 3×3 tile regions in the first captured image that surround each respective m×n pixel tile.

3. The method of claim 1 , wherein upsampling the bivariate surfaces uses an edge-aware filter on respective tile regions in the first captured image that surround each respective m×n pixel tile and are defined by one or more edges in the first captured image.

4. The method of claim 1 , wherein upsampling the bivariate surfaces uses a Kalman filter.

5. The method of claim 1 , wherein the first captured image and the second captured image were captured less than 1 second apart.

6. The method of claim 1 , wherein the m×n pixel tiles do not overlap with one another.

7. The method of claim 1 , wherein one or more entries in each distance matrix are respective minima, and wherein approximating the distance matrixes with respective bivariate surfaces comprises fitting minima of the respective bivariate surfaces to the respective minima of the distance matrixes.

8. The method of claim 7 , wherein fitting minima of the respective bivariate surfaces to the respective minima of the distance matrixes comprises fitting the respective bivariate surfaces to respective binomially-weighted 3×3 pixel regions surrounding the respective minima of the distance matrixes.

9. The method of claim 1 , wherein the m×n pixel tiles are 32×32 pixel tiles and the p×q pixel tiles are 64×64 pixel tiles.

10. The method of claim 1 , wherein a particular distance matrix for a particular m×n pixel tile is based on a linear combination of (i) a sum of squared values in the particular m×n pixel tile, (ii) squared values in the second captured image filtered by a box filter, and (iii) a cross-correlation of the second captured image and the particular m×n pixel tile.

11. The method of claim 1 wherein the bivariate surfaces are bivariate quadratic surfaces.

12. A camera device comprising:
an image sensor;
a processor;
memory; and
program instructions, stored in the memory, that upon execution cause the camera device to perform operations comprising:
capturing, by the image sensor, a first captured image of a scene and a second captured image of the scene;
for a plurality of m×n pixel tiles of the first captured image, determining, by the processor, respective distance matrixes, wherein the distance matrixes represent respective fit confidences between the m×n pixel tiles and pluralities of target p×q pixel tiles in the second captured image;
approximating, by the processor, the distance matrixes with respective bivariate surfaces;
upsampling, by the processor, the bivariate surfaces to obtain respective offsets for pixels in the plurality of m×n pixel tiles, such that the respective offsets, when applied to pixels in the plurality of m×n pixel tiles, cause parts of the first captured image to estimate locations in the second captured image; and
based on the first captured image, the second captured image, and the respective offsets, generating, by the processor, an intermediate image with intermediate locations of the pixels from the first captured image and the second captured image, wherein the intermediate image represents an intermediate version of the scene that is temporally or physically between those of the first captured image and the second captured image.

13. The camera device of claim 12 , wherein upsampling the bivariate surfaces uses bicubic interpolation of respective 3×3 tile regions in the first captured image that surround each respective m×n pixel tile.

14. The camera device of claim 12 , wherein upsampling the bivariate surfaces uses an edge-aware filter on respective tile regions in the first captured image that surround each respective m×n pixel tile and are defined by one or more edges in the first captured image.

15. The camera device of claim 12 , wherein the first captured image and the second captured image were captured less than 1 second apart.

16. The camera device of claim 12 , wherein one or more entries in each distance matrix are respective minima, and wherein approximating the distance matrixes with respective bivariate surfaces comprises fitting minima of the respective bivariate surfaces to the respective minima of the distance matrixes.

17. The camera device of claim 16 , wherein fitting minima of the respective bivariate surfaces to the respective minima of the distance matrixes comprises fitting the respective bivariate surfaces to respective binomially-weighted 3×3 pixel regions surrounding the respective minima of the distance matrixes.

18. The camera device of claim 12 , wherein a particular distance matrix for a particular m×n pixel tile is based on a linear combination of (i) a sum of squared values in the particular m×n pixel tile, (ii) squared values in the second captured image filtered by a box filter, and (iii) a cross-correlation of the second captured image and the particular m×n pixel tile.

19. A method comprising:
capturing, by an image sensor of a camera device, a first captured image of a scene and a second captured image of the scene;
for an m×n pixel tile of the first captured image, determining, by a processor of the camera device, a distance matrix, wherein the distance matrix represents fit confidences between the m×n pixel tile and a plurality of target p×q pixel tiles in the second captured image;
approximating, by the processor, the distance matrix with a bivariate surface;
upsampling, by the processor, the bivariate surface to obtain respective offsets for pixels in the m×n pixel tile, such that the respective offsets, when applied to pixels in the m×n pixel tile, cause parts of the first captured image to estimate locations in the second captured image; and
based on the first captured image, the second captured image, and the respective offsets, generating, by the processor, an intermediate image that with intermediate locations of the pixels from the first captured image and the second captured image, wherein the intermediate image represents an intermediate version of the scene that is temporally or physically between those of the first captured image and the second captured image.

2

Assignees

• Google Inc
• Google Llc

Inventors

• Barron Jonathan T
• Sharlet Dillon

Classifications

• G06T7/277 Primary

  involving stochastic approaches, e.g. using Kalman filters · CPC title

• H04N23/951

  by using two or more images to influence resolution, frame rate or aspect ratio · CPC title

• G06V10/759

  Region-based matching · CPC title

• G06V10/7515

  Shifting the patterns to accommodate for positional errors · CPC title

• G06T5/00

  Image enhancement or restoration · CPC title