Imaging device and method for generating an undistorted wide view image

Patent metadata

Field	Value
Publication number	US-9819865-B2
Application number	US-201615336588-A
Country	US
Kind code	B2
Filing date	Oct 27, 2016
Priority date	Oct 30, 2015
Publication date	Nov 14, 2017
Grant date	Nov 14, 2017

Abstract

Certain aspects of the technology disclosed herein involve combining images to generate a wide view image of a surrounding environment. Images can be recorded using an stand-alone imaging device having wide angle lenses and/or normal lenses. Images from the imaging device can be combined using methods described herein. In an embodiment, a pixel correspondence between a first image and a second image can be determined, based on a corresponding overlap area associated with the first image and the second image. Corresponding pixels in the corresponding overlap area associated with the first image and the second image can be merged based on a weight assigned to each of the corresponding pixels.

First claim

The invention claimed is:

1. A method to generate an undistorted 360° image of a surrounding environment, the method comprising:
receiving a first plurality of images corresponding to a first plurality of cameras, and a second plurality of images corresponding to a second plurality of cameras from a 360° imaging device, wherein at least one camera of the first plurality of cameras and the second plurality of cameras has a wide angle lens;
dewarping a distorted image of the first plurality of images or the second plurality of images, wherein the at least one camera having the wide angle lens recorded the distorted image;
determining a pixel correspondence between a first image in the first plurality of images and a second image in the second plurality of images, based on a corresponding overlap area associated with the first image and the second image; and
combining the first plurality of images and the second plurality of images by merging corresponding pixels in the corresponding overlap area, said merging comprising:
for each pixel in the first image associated with the corresponding overlap area, assigning a first weight, and wherein the first weight is associated with a determined distortion of a pixel in the first image; and
for each corresponding pixel in the second images associated with the corresponding overlap area, weighing the pixel by a second weight, wherein the second weight increases as the first weight decreases;
wherein the weight dictates a proportional contribution of each pixel to values of a merged pixel; and

tiling one or more non-overlapping areas and the corresponding overlap area to obtain a 360° image.

2. The method of claim 1 , further comprising:
obtaining a plurality of mapping functions corresponding to the at least one camera having the wide angle lens, wherein a mapping function in the plurality of mapping functions comprises a transformation, wherein when the transformation is applied to the distorted image associated with the wide angle lens, the transformation dewarping the distorted image and producing an undistorted image.

3. The method of claim 2 , wherein said obtaining the plurality of mapping functions comprises:
recording a first image using the wide angle lens;
recording a second image using a normal lens, wherein the normal lens is disposed the same way as the wide angle lens;
calculating the mapping function; and
storing the mapping function in a memory device.

4. The method of claim 2 , wherein said obtaining the plurality of mapping functions comprises retrieving the plurality of mapping functions from a memory device.

5. The method of claim 1 , wherein the said determining the pixel correspondence comprises:
identifying a plurality of overlap areas associated with the first image in the first plurality of images and the second image in the second plurality of images;
calculating a plurality of first match scores corresponding to the plurality of overlap areas;
identifying an overlap area of the plurality of overlap areas having a highest first match score of the plurality of first match scores as the corresponding overlap area;
identifying one or more first set of pixels in the corresponding overlap area of the first image and one or more second set of pixels in the corresponding overlap area of the second image;
calculating a plurality of second match scores corresponding to the one or more first set of pixels and the one or more second set of pixels; and
identifying the one or more first sets of pixels and the one or more second sets of pixels having a highest second match score of the plurality of second match scores as corresponding pixels.

6. The method of claim 1 , further comprising:
compressing the undistorted 360° image by storing image data of an identified object in a database and inserting a reference to the stored image data of the identified object in the database; and
encrypting the inserted reference to the stored image data of the identified object.

7. The method of claim 1 , further comprising:
identifying an object of a class in an image of the first plurality of images or the second plurality of images.

8. The method of claim 7 , further comprising:
performing any of:
removing the identified object from the image by merging pixels of the identified object with corresponding pixels of a corresponding overlap area, wherein the pixels of the identified object are assigned a weight of 0; or
locally dewarp a region surround the identified object by mapping corresponding pixels of the identified object in the first image and the second image to determine a distortion function and utilizing the distortion function to normalize the region surrounding the identified object.

9. The method of claim 1 , further comprising:
identifying an object of a class in an image of the first plurality of images or the second plurality of images recorded by a camera of the first plurality of cameras or the second plurality of cameras;
determining a location of the identified object in another image recorded by another camera; and
focusing on the determined location of the identified object in the another image recorded by the another camera.

10. The method of claim 1 , wherein merging corresponding pixels comprises evaluating weighted scores associated with a residual distortion of the corresponding pixels to determine a proportional contribution of each of the corresponding pixels.

11. An imaging device capable of recording an undistorted full view of a surrounding environment, the imaging device comprising:
a first plurality of cameras comprising a plurality of wide angle lenses, wherein the first plurality of cameras is configured to record a first plurality of images associated with the surrounding environment,
a second plurality of cameras comprising a plurality of normal lenses, wherein the second plurality of cameras is configured to record a second plurality of images associated with the surrounding environment; and
a processor configured to perform a method comprising:
obtain the first plurality of images and the second plurality of images;
dewarp the first plurality of images;
determine a pixel correspondence between a first image of the first plurality of images and a second image of the second plurality of images, based on a corresponding overlap area associated with the first image and the second image; and

combine the first plurality of images and the second plurality of images by merging corresponding pixels in the corresponding overlap area, said merging comprising:
for each pixel in the first image associated with the corresponding overlap area, assigning a first weight, and wherein the first weight is associated with a determined distortion of a pixel in the first image; and
for each corresponding pixel in the second images associated with the corresponding overlap area, weighing the pixel by a second weight, wherein the second weight increases as the first weight decreases;
wherein the weight dictates a proportional contribution of each pixel to values of a merged pixel; and
tiling one or more non-overlapping areas and the corresponding overlap area to obtain an undistorted full view image associated with the surrounding environment.

12. The imaging device of claim 11 , wherein the processor comprises any of:
a processor within the imaging device; and
a processor connected to the imaging device.

13. The imaging device of claim 11 , wherein determining the pixel correspondence comprises:
identifying a plurality of overlap areas associated with the first image in the first plurality of images and the second image in the second plural

Assignees

• Essential Products Inc

Inventors

• Evans V David John
• Jiang Xinrui
• Rubin Andrew E
• Hershenson Matthew
• Miao Xiaoyu
• Tate Joseph Anthony
• Gagne-Keats Jason Sean

Classifications

• H04N23/45

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title

• H04N23/698 Primary

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title

• G06F16/5866

  using information manually generated, e.g. tags, keywords, comments, manually generated location and time information · CPC title