Apparatus and method to record a 360 degree image

Patent metadata

Field	Value
Publication number	US-2017257566-A1
Application number	US-201715604544-A
Country	US
Kind code	A1
Filing date	May 24, 2017
Priority date	Oct 30, 2015
Publication date	Sep 7, 2017
Grant date	—

Abstract

Certain aspects of the technology disclosed herein combine wide angle lenses with normal lenses to create an undistorted 360° view of the surrounding environment. The normal lenses record an image of the surrounding environment in the areas where the wide angle lenses provide a distorted view. The wide angle lenses and the normal lenses can be part of a stand-alone imaging device, can be accessories to a mobile device, or can be integrated into a mobile device. Various ways to integrate the wide lenses and/or the normal lenses into the camera are disclosed herein.

First claim

1 . A method comprising:
determining a distortion associated with a wide angle lens by recording with the wide angle lens a distorted image of a first scene, and recording with a normal lens an image of the first scene, wherein the wide angle lens and the normal lens are oriented in substantially the same direction; recording a distorted image of a second scene with the wide angle lens, and an image of the second scene with the normal lens, by orienting the wide angle lens and the normal lens in different directions; and creating a wide field of view image by rectifying the distorted image of the second scene recorded by the wide angle lens based on the distortion associated with the wide angle lens, and combining the rectified image and the image of the second scene into the wide field of view image.

2 . The method of claim 1 , wherein said determining the distortion comprises determining a transformation between the distorted image of the first scene and the image of the first scene, wherein applying the transformation to the distorted image of the first scene produces substantially the image of the first scene.

3 . The method of claim 1 , said recording the wide field of view image comprising orienting the wide angle lens in a first direction, and orienting the normal lens in a second direction substantially 90° away from the first direction.

4 . The method of claim 1 , said combining the rectified image and the image of the second scene into the wide field of view image comprising determining a correspondence region between the rectified image and the image of the second scene.

5 . The method of claim 4 , said determining the correspondence region comprising:
creating an overlap region between the rectified image and the image of the second scene by establishing a correspondence between a first pixel in the rectified image and a first pixel in the image of the second scene; calculating a difference between the first pixel in the rectified image and the first pixel in the image of the second scene; reducing the overlap region until the overlap region is empty; and determining the correspondence region to be the overlap region associated with the smallest difference between the first pixel in the rectified image and the first pixel in the image of the second scene.

6 . The method of claim 5 , said calculating the difference comprising calculating a least square difference between the first pixel in the rectified image and the first pixel in the image of the second scene.

7 . The method of claim 4 , comprising blending pixels in the correspondence region between the rectified image and the image of the second scene.

8 . The method of claim 7 , said blending pixels comprising:
modulating a pixel in the correspondence region associated with the rectified image by a first weight; modulating the pixel in the correspondence region associated with the image of the second scene by a second weight; and adding the modulated first pixel in the correspondence region and the modulated first pixel in the image of the second scene to produce a pixel in the wide field of view image.

9 . The method of claim 8 , wherein the first weight and the second weight add up to a constant number.

10 . A method comprising:
obtaining a plurality of mapping functions corresponding to a plurality of wide angle lenses, wherein a mapping function in the plurality of mapping functions comprises a transformation, wherein when the transformation is applied to a distorted image associated with a wide angle lens in the plurality of wide angle lenses the transformation produces an undistorted image; receiving a first plurality of images corresponding to a first plurality of cameras, and a second plurality of images corresponding to a second plurality of cameras, wherein at least one image in the first plurality of images and the second plurality of images comprises the distorted image; determining a pixel correspondence between the first image in the first plurality of images and the second image in the second plurality of images, based on an overlap area associated with the first image and the second image; and combining the first plurality of undistorted images and the second plurality of undistorted images to obtain an undistorted 360° image.

11 . The method of claim 10 , wherein said obtaining the plurality of mapping functions comprises:
recording a first image using the wide angle lens; recording a second image using a normal lens, wherein the normal lens is disposed the same way as the wide angle lens; calculating the mapping function; and storing the mapping function in a memory.

12 . The method of claim 10 , wherein said obtaining the plurality of mapping functions comprises retrieving the plurality of mapping functions from a memory.

13 . The method of claim 10 , wherein the said determining the pixel correspondence comprises:
applying the mapping function to each distorted image, to obtain a first plurality of undistorted images, and second plurality of undistorted images, the first plurality of undistorted images corresponding to the first plurality of cameras, and the second plurality of undistorted images corresponding to the second plurality of cameras; iteratively overlapping each image in the first plurality of undistorted images and a corresponding image in the second plurality of undistorted images to obtain a plurality of overlap areas, wherein the overlap area in the plurality of overlap areas is associated with each iteration; iteratively calculating a plurality of match scores corresponding to a plurality of overlap areas between each image in the first plurality of undistorted images and the corresponding image in the second plurality of undistorted images; and determining the pixel correspondence based on a best overlap area in the plurality of overlap areas, wherein the best overlap area is associated with the highest match score in the plurality of match scores.

14 . The method of claim 10 , wherein said combining the first plurality of undistorted images and the second plurality of undistorted images comprises:
obtaining the overlap area associated with the first image in the first plurality of images, and the second image and the second plurality of images; determining a first portion of the first image in the first plurality of image, the first portion outside the overlap area; determining a second portion of the second image in the second plurality of images, the second portion outside the overlap area; determining an overlap portion, said determining comprising:
for each pixel in the first image associated with the overlap area, weighing the pixel by a first weight, wherein the first weight comprises a number between 0 and 1 inclusive, and wherein the first weight decreases with an increasing distance from an edge associated with the overlap area;
for each corresponding pixel in the second image associated with the overlap area, weighing the pixel by a second weight, wherein the second weight increases as the first weight decreases;
adding the two pixels together to obtain a pixel in the overlap portion; and
tiling the first portion, the second portion, and the overlap portion to obtain a 360° image.

15 . A system comprising:
a wide angle lens; a normal lens; a processor; a storage medium storing computer-executable instructions that, when executed by the processor, cause the system to perform a computer-implemented operation, the instructions comprising:
instructions for determining a distortion associated with the wide angle lens by recording with the wide angle l

Assignees

• Essential Products Inc

Inventors

• Evans V David John
• Jiang Xinrui
• Rubin Andrew E
• Hershenson Matthew
• Miao Xiaoyu

Classifications

• H04N23/55

  Optical parts specially adapted for electronic image sensors; Mounting thereof · CPC title

• H04N23/51

  Housings · CPC title

• H04N23/62

  Control of parameters via user interfaces · CPC title

• H04N23/698 Primary

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title