Method for combining images

Patent metadata

Field	Value
Publication number	US-9600863-B2
Application number	US-201213371737-A
Country	US
Kind code	B2
Filing date	Feb 13, 2012
Priority date	Feb 13, 2012
Publication date	Mar 21, 2017
Grant date	Mar 21, 2017

Abstract

A method for combining images includes capturing a first image including a subject from a first camera. A second image is captured from a second camera and the second image includes the subject. First pre-processing functions are applied on the first image to produce a first processed image. The first pre-processing functions include applying a distortion component of a rotation matrix to the first image. The rotation matrix defines a corrected relationship between the first and the second image. Second pre-processing functions are applied on the second image to produces a second processed image. The second pre-processing functions include applying the rotation matrix to the second image. The first processed image and the second processed image are blended in a processing unit to form a composite image.

First claim

What is claimed is:

1. A method for combining images, the method comprising:
capturing a first image including a subject from a first camera;
capturing a second image including the subject from a second camera;
applying first pre-processing functions on the first image to produce a first processed image, the first pre-processing functions including applying a distortion component of a rotation matrix to the first image, wherein the rotation matrix defines a corrected relationship between the first and the second image;
applying second pre-processing functions on the second image to produce a second processed image, the second pre-processing functions including applying the rotation matrix to the second image; and
blending, in a processing unit, the first processed image and the second processed image to form a composite image different from the first and second images captured by the first and second cameras;
wherein the distortion component and the rotation matrix are pre-determined before capturing the first image and the second image by:
capturing a first calibration image including a calibration subject from the first camera before capturing the first image and the second image;
capturing a second calibration image including the calibration subject from the second camera before capturing the first image and the second image;
identifying corresponding pixel sets, each corresponding pixel set including a first pixel in the first calibration image that corresponds with a second pixel in the second calibration image, wherein the first pixel and the second pixel mark the same point on the calibration subject; and
determining the distortion component and a rotation component of the rotation matrix by minimizing a distance between the first pixel in the first calibration image and the second pixel in the second calibration image in each of the corresponding pixel sets, wherein the distortion component and the rotation component are determined in an iterative loop until the distance is converged, and wherein the rotation matrix is the product of the distortion component and the rotation component;

wherein the first camera and the second camera have substantially identical image sensors and optics and are positioned at a known rotation angle, wherein the optics of the first camera and the second camera include a wide-angle lens, and wherein the distortion component corrects for the wide-angle lens; and
wherein during calibration the rotation component is kept fixed while an optimized distortion component is determined and the optimized distortion component is kept fixed and used to determine an optimized rotation component.

2. The method of claim 1 , wherein the first pre-processing functions include converting the first image to a cylindrical coordinate system, and wherein the second pre-processing functions include converting the second image to the cylindrical coordinate system.

3. The method of claim 1 , wherein a distortion value representing the distortion component of the rotation matrix and a rotation matrix value representing the rotation matrix are stored in a memory communicatively coupled to a processor disposed in the processing unit.

4. The method of claim 1 , wherein blending the first processed image and the second processed image to form the composite image includes shifting all the pixels in the processed second image a defined distance, causing a portion of the pixels in the second processed image to overlap the first processed image.

5. The method of claim 1 , wherein applying the rotation matrix to the second image includes multiplying the rotation matrix and each pixel in the second image to achieve real-time processing of the composite image.

6. The method of claim 1 , further comprising storing the composite image to a memory communicatively coupled to a processor disposed in the processing unit and rendering the composite image to a display device.

7. The method of claim 1 , wherein the distortion component and the rotation component are calculated using Nelder-Mead Simplex Method.

8. The method of claim 1 , wherein a portion of the first processed image overwrites in a memory a portion of the second processed image to form the composite image.

9. The method of claim 1 wherein the optimized distortion component is used as the distortion component, the optimized rotation component is used as the rotation component, and the optimized distortion component and the optimized rotation component are multiplied to form the rotation matrix.

10. A non-transitory machine-accessible storage medium that provides instructions that, when executed by a computer, will cause the computer to perform operations comprising:
receiving a first image including a subject captured from a first camera;
receiving a second image including the subject captured from a second camera;
applying first pre-processing functions on the first image to produce a first processed image, the first pre-processing functions including applying a distortion component of a rotation matrix to the first image, wherein the rotation matrix defines a corrected relationship between the first and the second image;
applying second pre-processing functions on the second image to produce a second processed image, the second pre-processing functions including applying the rotation matrix to the second image; and
blending, in a processing unit, the first processed image and the second processed image to form a composite image;
wherein the distortion component and the rotation matrix are pre-determined before capturing the first image and the second image by:
capturing a first calibration image including a calibration subject from the first camera before capturing the first image and the second image;
capturing a second calibration image including the calibration subject from the second camera before capturing the first image and the second image;
identifying corresponding pixel sets, each corresponding pixel set including a first pixel in the first calibration image that corresponds with a second pixel in the second calibration image, wherein the first pixel and the second pixel mark the same point on the calibration subject; and
determining the distortion component and the rotation component of the rotation matrix by minimizing a distance between the first pixel in the first calibration image and the second pixel in the second calibration image in each of the corresponding pixel sets, wherein the distortion component and the rotation component are determined in an iterative loop until the distance is converged, and wherein the rotation matrix is the product of the distortion component and the rotation component;

wherein the first camera and the second camera have substantially identical image sensors and optics and are positioned at a known rotation angle, wherein the optics of the first camera and the second camera include a wide-angle lens, and wherein the distortion component corrects for the wide-angle lens; and
wherein during calibration the rotation component is kept fixed while an optimized distortion component is determined and the optimized distortion component is kept fixed and used to determine an optimized rotation component.

11. The non-transitory machine-accessible storage medium of claim 10 , wherein the first pre-processing functions include converting the first image to a cylindrical coordinate system, and wherein the second pre-processing functions include converting the second image to the cylindrical coordinate system.

12. The non-transitory machine-accessible storage medium of claim 10 , wherein a distortion value representing the distortion component of the rotation matrix

Assignees

• Xiong Weihua
• Shan Jizhang
• Wu Donghui
• Omnivision Tech Inc

Inventors

• Xiong Weihua
• Shan Jizhang
• Wu Donghui

Classifications

• G06T3/60 Primary

  Rotation of whole images or parts thereof · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• G06T2207/30208

  Marker matrix · CPC title

• G06T7/85

  Stereo camera calibration · CPC title

• G06T7/33

  using feature-based methods · CPC title