Stereo-motion method of three-dimensional (3-D) structure information extraction from a video for fusion with 3-D point cloud data

Patent metadata

Field	Value
Publication number	US-9761002-B2
Application number	US-201313953758-A
Country	US
Kind code	B2
Filing date	Jul 30, 2013
Priority date	Jul 30, 2013
Publication date	Sep 12, 2017
Grant date	Sep 12, 2017

Abstract

According to an embodiment, a method for generating a 3-D stereo structure comprises registering and rectifying a first image frame and a second image frame by local correction matching, extracting a first scan line from the first image frame, extracting a second scan line from the second image frame corresponding to the first scan line, calculating a pixel distance between the first scan line and the second scan line for each pixel for a plurality of pixel shifts, calculating a smoothed pixel distance for each pixel for the pixel shifts by filtering the pixel distance for each pixel over the pixel shifts, and determining a scaled height for each pixel of the first scan line, the scaled height comprising a pixel shift from among the pixel shifts corresponding to a minimal distance of the smoothed pixel distance for the pixel.

First claim

The invention claimed is:

1. A method, comprising:
generating a three-dimensional (3-D) stereo structure from a first image frame and a second image frame by at least:
rectifying the first image frame and the second image frame by local correction matching, wherein the local correction matching comprises determining a plurality of correlation coefficient values between a first pixel selected from the first image frame and a plurality of second pixels selected within a template area of the second image frame, and wherein the template area is smaller than the second image frame;
extracting a first scan line from the first image frame, the first scan line comprising a first plurality of pixels;
extracting a second scan line from the second image frame corresponding to the first scan line, the second scan line comprising a second plurality of pixels;
calculating a first sequence of pixel distances between corresponding pixels in the first plurality of pixels and the second plurality of pixels;
shifting the second plurality of pixels by a shift value that is greater than zero;
calculating a second sequence of pixel distances between corresponding pixels in the first plurality of pixels and the shifted second plurality of pixels;
determining a minimal distance for a particular pixel of the first scan line from among at least the first sequence of pixel distances and the second sequence of pixel distances; and
determining a scaled height for the particular pixel of the first scan line, the scaled height comprising a pixel shift corresponding to the minimal distance for the particular pixel.

2. The method of claim 1 , further comprising determining one or more shapes in each of the first image frame and the second image frame.

3. The method of claim 1 , wherein the 3-D stereo structure is used to locate a position of an unmanned aerial vehicle.

4. The method of claim 1 , wherein rectifying the first image frame and the second image frame comprises:
determining a first shape in the first image frame, wherein the first shape has a consistent color;
matching a second shape in the second image frame with the first shape; and
determining one or more matching points between the first shape and the second shape.

5. The method of claim 1 , wherein the first image frame and the second image frame comprise frames from a full motion video.

6. The method of claim 5 , further comprising generating a stereo structure from the full motion video.

7. A system, comprising:
a processor; and
data storage configured to store at least computer-readable instructions that, when executed by the processor, cause the system to perform functions, the functions comprising:
generating a three-dimensional (3-D) stereo structure from a first image frame and a second image frame by at least:
rectifying the first image frame and the second image frame by local correction matching, wherein the local correction matching comprises determining a plurality of correlation coefficient values between a first pixel selected from the first image frame and a plurality of second pixels selected within a template area of the second image frame, and wherein the template area is smaller than the second image frame;
extracting a first scan line from the first image frame, the first scan line comprising a first plurality of pixels;
extracting a second scan line from the second image frame corresponding to the first scan line, the second scan line comprising a second plurality of pixels;
calculating a first sequence of pixel distances between corresponding pixels in the first plurality of pixels and the second plurality of pixels;
shifting the second plurality of pixels by a shift value that is greater than zero;
calculating a second sequence of pixel distances between corresponding pixels in the first plurality of pixels and the shifted second plurality of pixels;
determining a minimal distance for a particular pixel of the first scan line from among at least the first sequence of pixel distances and the second sequence of pixel distances; and
determining a scaled height for the particular pixel of the first scan line, the scaled height comprising a pixel shift corresponding to the minimal distance for the particular pixel.

8. The system of claim 7 , wherein the functions further comprise:
determining one or more shapes in each of the first image frame and the second image frame.

9. The system of claim 7 , wherein the 3-D stereo structure is used to locate a position of an unmanned aerial vehicle.

10. The system of claim 7 , wherein rectifying the first image frame and the second image frame comprises:
determining a first shape in the first image frame, wherein the first shape has a consistent color;
matching a second shape in the second image frame with the first shape; and
determining one or more matching points between the first shape and the second shape.

11. The system of claim 7 , wherein the first image frame and the second image frame comprise frames from a full motion video.

12. The system of claim 11 , wherein the functions further comprise:
generating a stereo structure from the full motion video.

13. The system of claim 12 , wherein the stereo structure is used to locate a position of an unmanned aerial vehicle.

14. A non-transitory computer readable storage medium comprising computer-executable instructions that, when executed by a processor of a system, cause the system to perform functions comprising:
generating a three-dimensional (3-D) stereo structure from a first image frame and a second image frame by at least:
rectifying the first image frame and the second image frame by local correction matching, wherein the local correction matching comprises determining a plurality of correlation coefficient values between a first pixel selected from the first image frame and a plurality of second pixels selected within a template area of the second image frame, and wherein the template area is smaller than the second image frame;
extracting a first scan line from the first image frame, the first scan line comprising a first plurality of pixels;
extracting a second scan line from the second image frame corresponding to the first scan line, the second scan line comprising a second plurality of pixels;
calculating a first sequence of pixel distances between corresponding pixels in the first plurality of pixels and the second plurality of pixels;
shifting the second plurality of pixels by a shift value that is greater than zero;
calculating a second sequence of pixel distances between corresponding pixels in the first plurality of pixels and the shifted second plurality of pixels;
determining a minimal distance for a particular pixel of the first scan line from among at least the first sequence of pixel distances and the second sequence of pixel distances; and
determining a scaled height for the particular pixel of the first scan line, the scaled height comprising a pixel shift corresponding to the minimal distance for the particular pixel.

15. The non-transitory computer readable storage medium of claim 14 , the functions further comprising determining one or more shapes in each of the first image frame and the second image frame.

16. The non-transitory computer readable storage medium of claim 14 , wherein the 3-D stereo structure is used to locate a position of an unmanned aerial vehicle.

17. The non-transitory computer readable storage medium of claim 14 , wherein rectifying the first image frame and the second image frame comprises:
determining a first shape in the first

Assignees

• Boeing Co

Inventors

• Mundhenk Terrell Nathan
• Chen Hai-Wen
• Owechko Yuri
• Korchev Dmitriy
• Kim Kyungnam
• Zhang Zhiqi

Classifications

• G06T2207/10012

  Stereo images · CPC title

• G06T2207/30181

  Earth observation · CPC title

• G06T2207/10032

  Satellite or aerial image; Remote sensing · CPC title

• G06T7/593 Primary

  from stereo images · CPC title

• G06T7/0075 Primary

  Physics · mapped topic