Color reconstruction

Patent metadata

Field	Value
Publication number	US-9961283-B2
Application number	US-201615284042-A
Country	US
Kind code	B2
Filing date	Oct 3, 2016
Priority date	Sep 7, 2016
Publication date	May 1, 2018
Grant date	May 1, 2018

Abstract

In one embodiment, coloring artifacts of a color image output by a camera are minimized by taking into account a distortion introduced by the lens. Based on the distortion, the color reconstruction determines which pixels in the grayscale image to include in the reconstruction process. Additionally, the color reconstruction can take into account edges depicted in the grayscale image to determine which pixels to include in the reconstruction process. In another embodiment, coloring artifacts in a 360 degree color image are minimized by performing the color reconstruction process on a three-dimensional surface. Before the color reconstruction takes place, the two-dimensional grayscale image is projected onto a three-dimensional surface, and the color reconstruction is performed on the three-dimensional surface. The color reconstruction on the three-dimensional surface can take into account the distortion produced by the lens and/or can take into account the edges depicted in the two-dimensional and three-dimensional grayscale image.

First claim

The invention claimed is:

1. A method to generate a 360 degree color image in a spherical space, based on a two-dimensional grayscale image and a color designation comprising:
obtaining from a 360 degree camera the two-dimensional grayscale image and the color designation, wherein the color designation specifies one or more colors associated with the two-dimensional grayscale image;
mapping the two-dimensional grayscale image onto a representation of a sphere to obtain a three-dimensional grayscale image, while preserving a correspondence between the color designation and the three-dimensional grayscale image, said projecting comprising:
applying a transformation to a plurality of pixels in the two-dimensional grayscale image to generate a pixel in the three-dimensional grayscale image, wherein the transformation comprises a plurality of weights corresponding to the plurality of pixels in the two-dimensional grayscale image, wherein the plurality of weights add up to a constant number;
storing the transformation to preserve the correspondence between the color designation and the 3-dimensional grayscale image; and

based on the three-dimensional grayscale image and the correspondence between the color designation and the three-dimensional grayscale image, generating the 360 degree color image.

2. The method of claim 1 , said generating the 360 degree color image comprising:
determining a distortion of the pixel in the three-dimensional grayscale image based on a distortion model associated with the 360 degree camera;
calculating, based on the distortion associated with the pixel, a pixel neighborhood size of a pixel neighborhood proximate to the pixel in the three-dimensional grayscale image, wherein said calculating includes negatively correlating the pixel neighborhood size to a magnitude of the distortion; and
calculating a color associated with the pixel in the three-dimensional grayscale image based on the pixel neighborhood size, to generate the 360 color image.

3. A method comprising:
obtaining from a 360 degree camera a two-dimensional grayscale image and a color designation, wherein the color designation specifies one or more colors associated with the two-dimensional grayscale image;
mapping the two-dimensional grayscale image onto a representation of a three-dimensional surface to obtain a three-dimensional grayscale image, while preserving a correspondence between the color designation and the three-dimensional grayscale image; and
based on the three-dimensional grayscale image and the correspondence between the color designation and the three-dimensional grayscale image, generating a 360 degree color image.

4. The method of claim 3 , wherein the correspondence between the color designation and the three-dimensional grayscale image comprises a transformation such that when the transformation is applied to a plurality of pixels in the two-dimensional grayscale image, a pixel in the three-dimensional grayscale image is generated.

5. The method of claim 4 , wherein the transformation comprises a plurality of weights corresponding to the plurality of pixels in the two-dimensional grayscale image, wherein the plurality of weights add up to a constant number.

6. The method of claim 5 , said generating the 360 degree color image comprising:
based on the plurality of weights associated with the transformation, determining a color deficiency associated with the pixel in the three-dimensional grayscale image, wherein the color deficiency corresponds to each color in the color designation; and
based on the color deficiency, calculating a color associated with the pixel in the three-dimensional grayscale image.

7. The method of claim 3 , said projecting the two-dimensional grayscale image onto the three-dimensional surface to obtain the three-dimensional grayscale image, while preserving the correspondence between the color designation and the three-dimensional grayscale image comprising:
determining an edge depicted in the two-dimensional grayscale image;
determining a transformation associated with a pixel in the three-dimensional grayscale image, and a plurality of pixels in the two-dimensional grayscale image to which to apply the transformation, such that the plurality of pixels in the two-dimensional grayscale image do not span the edge depicted in the two-dimensional grayscale image; and
assigning the transformation to the correspondence between the color designation and the three-dimensional grayscale image.

8. The method of claim 7 , wherein the transformation comprises a plurality of weights corresponding to the plurality of pixels in the two-dimensional grayscale image, wherein the plurality of weights add up to a constant number.

9. The method of claim 8 , said generating the 360 degree color image comprising:
based on the plurality of weights associated with the transformation, determining a color deficiency associated with the pixel in the three-dimensional grayscale image, wherein the color deficiency corresponds to each color in the color designation; and
based on the color deficiency, calculating a color associated with the pixel in the three-dimensional grayscale image.

10. The method of claim 3 , wherein the three-dimensional surface corresponds to a surface of a lens associated with the 360 degree camera.

11. The method of claim 3 , wherein the three-dimensional surface comprises an ellipsoid.

12. The method of claim 3 , said generating the 360 degree color image comprising:
determining a distortion of a pixel in the three-dimensional grayscale image based on a distortion model associated with the 360 degree camera;
calculating, based on the distortion associated with the pixel, a pixel neighborhood size of a pixel neighborhood proximate to the pixel in the three-dimensional grayscale image, wherein said calculating includes negatively correlating the pixel neighborhood size to a magnitude of the distortion; and
calculating a color associated with the pixel in the three-dimensional grayscale image based on the pixel neighborhood size, to generate the 360 degree color image.

13. The method of claim 12 , wherein determining the distortion comprises calculating the distortion based on a distance between the pixel and a center associated with the three-dimensional grayscale image.

14. The method of claim 12 , wherein determining the distortion comprises accessing a distortion mapping table associated with the 360 degree camera.

15. The method of claim 12 , further comprising:
determining an edge depicted in the three-dimensional grayscale image; and
determining the pixel neighborhood such that a path between the pixel and each pixel associated with the pixel neighborhood does not cross the edge depicted in the three-dimensional grayscale image, and a distance between the pixel and each pixel associated with the pixel neighborhood is within the pixel neighborhood size.

16. The method of claim 15 , said calculating the color associated with the pixel comprising:
for each pixel in the pixel neighborhood, assigning a weight to each pixel, such that a sum of weights associated with the pixel neighborhood does not exceed a constant number;
for each pixel in the pixel neighborhood, multiplying the color associated with each pixel by the weight to obtain an addend; and
summing addends associated with the pixel neighborhood to calculate the color associated with the pixel.

17. A system to generate a 360 degree color image in a spherical space, based on a two-dimensional grayscale image and a color designation comprising:
a 360 degree camer

Assignees

• Essential Products Inc

Inventors

• Kolb Michael
• Rubin Andrew E
• Hershenson Matthew
• Zavin Rebecca Schultz

Classifications

• H04N23/841

  to modify gamut · CPC title

• H04N23/12

  with one sensor only · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N25/702

  SSIS architectures characterised by non-identical, non-equidistant or non-planar pixel layout · CPC title

• H04N25/615

  involving a transfer function modelling the optical system, e.g. optical transfer function [OTF], phase transfer function [PhTF] or modulation transfer function [MTF] · CPC title