Color reconstruction

Patent metadata

Field	Value
Publication number	US-9800852-B1
Application number	US-201615284031-A
Country	US
Kind code	B1
Filing date	Oct 3, 2016
Priority date	Sep 7, 2016
Publication date	Oct 24, 2017
Grant date	Oct 24, 2017

Abstract

In one embodiment, coloring artifacts of a color image output by a camera are minimized by taking into account a distortion introduced by the lens. Based on the distortion, the color reconstruction determines which pixels in the grayscale image to include in the reconstruction process. Additionally, the color reconstruction can take into account edges depicted in the grayscale image to determine which pixels to include in the reconstruction process. In another embodiment, coloring artifacts in a 360 degree color image are minimized by performing the color reconstruction process on a three-dimensional surface. Before the color reconstruction takes place, the two-dimensional grayscale image is projected onto a three-dimensional surface, and the color reconstruction is performed on the three-dimensional surface. The color reconstruction on the three-dimensional surface can take into account the distortion produced by the lens and/or can take into account the edges depicted in the two-dimensional and three-dimensional grayscale image.

First claim

The invention claimed is:

1. A method to reconstruct a color associated with a grayscale image across pixels lacking the color, the method comprising:
obtaining from an imaging device the grayscale image and a color designation, wherein each pixel associated with the grayscale image comprises a grayscale value, and wherein the color designation specifies a single color associated with the grayscale value;
determining a distortion of a target pixel in the grayscale image based on a distortion model associated with the imaging device;
calculating, based on the distortion associated with the target pixel, a pixel neighborhood and a pixel neighborhood size of the pixel neighborhood proximate to the target pixel in the grayscale image, wherein said calculating includes negatively correlating the pixel neighborhood size to a magnitude of the distortion;
calculating a color associated with the target pixel in the grayscale image based on the pixel neighborhood size; and
based on the color associated with the target pixel, generating a color image, wherein a resolution associated with the color image is the same as a resolution associated with the grayscale image.

2. The method of claim 1 , wherein determining the distortion comprises calculating the distortion based on a distance between the target pixel and a center associated with the grayscale image.

3. The method of claim 1 , wherein determining the distortion comprises accessing a distortion mapping table associated with the imaging device.

4. The method of claim 1 , said calculating the pixel neighborhood comprising:
determining an edge depicted in the grayscale image; and
determining the pixel neighborhood such that a path between the target pixel and each pixel associated with the pixel neighborhood does not cross the edge depicted in the grayscale image, and a distance between the target pixel and each pixel associated with the pixel neighborhood is within the pixel neighborhood size.

5. The method of claim 4 , said calculating the color associated with the target pixel comprising:
for each pixel in the pixel neighborhood, assigning a weight to each pixel, such that a sum of weights associated with the pixel neighborhood does not exceed a constant number;
for each pixel in the pixel neighborhood, multiplying the color associated with each pixel by the weight to obtain an addend; and
summing addends associated with the pixel neighborhood to calculate the color associated with the target pixel.

6. A method comprising:
determining a distortion associated with a pixel in a grayscale image taken through a distortion-causing lens based on a distortion model associated with the distortion-causing lens;
calculating, based on the distortion associated with the pixel, a pixel neighborhood size of a pixel neighborhood proximate to the pixel in the grayscale image, wherein said calculating includes negatively correlating the pixel neighborhood size to a magnitude of the distortion; and
calculating a color associated with the pixel in the grayscale image based on the pixel neighborhood size to generate a color image.

7. The method of claim 6 , wherein determining the distortion comprises calculating the distortion based on one or more input parameters to a distortion function associated with the distortion-causing lens.

8. The method of claim 7 , wherein the one or more input parameters comprises a location of the pixel associated with the grayscale image, and the distortion function comprises a distance to a center associated with the grayscale image.

9. The method of claim 7 , wherein the one or more input parameters comprises a location of the pixel associated with the grayscale image, and the distortion function comprises a distance to an edge associated with the grayscale image.

10. The method of claim 6 , wherein determining the distortion comprises accessing a distortion mapping table associated with the distortion-causing lens.

11. The method of claim 6 , further comprising:
projecting the grayscale image onto a three-dimensional surface.

12. The method of claim 6 , further comprising:
determining an edge depicted in the grayscale image; and
determining the pixel neighborhood such that a path between the pixel and each pixel associated with the pixel neighborhood does not cross the edge depicted in the grayscale image, and a distance between the pixel and each pixel associated with the pixel neighborhood is within the pixel neighborhood size.

13. The method of claim 12 , said calculating the color associated with the pixel comprising:
obtaining a color designation associated with each pixel in the grayscale image;
for each pixel in the pixel neighborhood, assigning a weight to each pixel, such that a sum of weights associated with the pixel neighborhood does not exceed a constant number;
for each pixel in the pixel neighborhood, multiplying the color associated with each pixel by the weight to calculate an addend; and
summing addends associated with the pixel neighborhood to calculate the color associated with the pixel.

14. The method of claim 6 , further comprising determining the pixel neighborhood such that a distance between the pixel and each pixel associated with the pixel neighborhood is within the pixel neighborhood size.

15. The method of claim 6 , said calculating the color associated with the pixel comprising:
obtaining color designation associated with each pixel in the grayscale image;
for each pixel in the pixel neighborhood, assigning a weight to each pixel, such that a sum of weights associated with the pixel neighborhood does not exceed a constant number;
for each pixel in the pixel neighborhood, multiplying the color associated with each pixel by the weight to obtain an addend; and
summing addends associated with the pixel neighborhood to calculate the color associated with the pixel.

16. A system to reconstruct a color associated with a grayscale image across pixels lacking the color comprising:
an imaging device comprising a distortion-causing lens, wherein the distortion-causing lens produces the grayscale image;
a processor coupled to the imaging device;
a storage medium storing computer-executable instructions that, when executed by the processor, cause the system to perform a computer-implemented operation, the instructions comprising:
instructions for obtaining from the imaging device the grayscale image and a color designation, wherein each pixel associated with the grayscale image comprises a grayscale value, and wherein the color designation associates the grayscale value with a single color;
instructions for determining a distortion of a pixel in the grayscale image based on a distortion model associated with the imaging device;
instructions for calculating, based on the distortion associated with the pixel, a pixel neighborhood size of a pixel neighborhood proximate to the pixel in the grayscale image, wherein said calculating includes negatively correlating the pixel neighborhood size to a magnitude of the distortion; and
instructions for calculating a color associated with the pixel in the grayscale image based on the pixel neighborhood size, to generate a color image.

17. The system of claim 16 , wherein instructions for determining the distortion comprise instructions for calculating the distortion based on a distance between the pixel and a center associated with the grayscale image.

18. The system of claim 16 , wherein instructions for determining the distortion comprise instructions for accessing a distortion mapping table associat

Assignees

• Essential Products Inc

Inventors

• Kolb Michael
• Rubin Andrew E
• Hershenson Matthew
• Zavin Rebecca Schultz

Classifications

• H04N23/841

  to modify gamut · CPC title

• H04N25/615

  involving a transfer function modelling the optical system, e.g. optical transfer function [OTF], phase transfer function [PhTF] or modulation transfer function [MTF] · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/12

  with one sensor only · CPC title

• H04N25/702

  SSIS architectures characterised by non-identical, non-equidistant or non-planar pixel layout · CPC title