Stereo vision for sensing vehicles operating environment

Patent metadata

Field	Value
Publication number	US-9819925-B2
Application number	US-201514689881-A
Country	US
Kind code	B2
Filing date	Apr 17, 2015
Priority date	Apr 18, 2014
Publication date	Nov 14, 2017
Grant date	Nov 14, 2017

Abstract

A vehicle including a chassis, a drive system carrying the chassis, and a vision system carried by the chassis. The vision system having a stereo visible light camera producing a colorized 3D point cloud and a stereo long wave infrared camera producing 3D data. The vision system being configured to fuse the 3D data with the 3D point cloud thereby producing an enhanced 3D point cloud.

First claim

What is claimed is:

1. A vehicle, comprising:
a chassis;
a drive system carrying the chassis; and
a vision system carried by said chassis, said vision system including:
a stereo visible light camera producing a colorized 3D data point cloud;
a stereo long wave infrared (LWIR) camera producing 3D LWIR data; and
a near infrared (NIR) camera, wherein both said LWIR camera and said NIR camera produce data that is fused with said 3D point cloud,
wherein said fusing includes detecting foliage by way of measuring and comparing NIR and red spectrum energy level ratios,
wherein the LWIR camera data is used to fill in information in areas of said 3D point cloud including areas where there is low light at night or dust which obscures a field of vision of said stereo visible light camera, and wherein sparse computations are used to determine sparse points based on edge features, and wherein a segmentation computation and a hole-filling computation are used to fill in regions, including depth regions, between sparse points to fuse with the 3D point cloud;
wherein said vision system is configured to fuse said 3D data, detected data, and filled-in data with said 3D point cloud to produce an enhanced 3D point cloud.

2. The vehicle of claim 1 , wherein said drive system is directed to at least one of steer, change velocity, start and stop the vehicle, dependent upon said enhanced 3D point cloud.

3. The vehicle of claim 1 , wherein said vision system is further configured to extract features from the 3D point cloud and the 3D data as part of producing said enhanced 3D point cloud.

4. The vehicle of claim 3 , wherein said vision system is further configured to match at least some of the extracted features from the 3D point cloud and the 3D data as part of producing said enhanced 3D point cloud.

5. The vehicle of claim 4 , wherein said vision system is further configured to perform a consistency validation of the 3D data that is matched.

6. A vision system for use by a vehicle having a drive system, the vision system comprising:
a stereo visible light camera producing a colorized 3D point cloud;
a stereo long wave infrared (LWIR) camera producing 3D LWIR data; and
a near infrared (NIR) camera, wherein both said stereo LWIR camera and said NIR camera produce data that is fused with said 3D point cloud;
wherein said fusing includes detecting foliage by way of measuring and comparing NIR and red spectrum energy level ratios,
wherein LWIR data is used to fill in information in areas of said 3D point including areas where there is low light at night or dust which obscures a field of vision of said stereo visible light camera, and wherein sparse computations are used to determine sparse points based on edge features, and wherein a segmentation computation and a hole-filling computation are used to fill in regions, including depth regions, between sparse points to fuse with the 3D point cloud,
wherein said vision system is configured to fuse said 3D LWIR data, detected data, and filled-in data with said 3D point cloud to produce an enhanced 3D point cloud.

7. The vision system of claim 6 , wherein the drive system is directed to at least one of steer, change velocity, start and stop the vehicle, dependent upon said enhanced 3D point cloud.

8. The vision system of claim 6 , wherein said vision system is further configured to extract features from the 3D point cloud and the 3D data as part of producing said enhanced 3D point cloud.

9. The vision system of claim 8 , wherein said vision system is further configured to match at least some of the extracted features from the 3D point cloud and the 3D data as part of producing said enhanced 3D point cloud.

10. The vision system of claim 9 , wherein said vision system is further configured to perform a consistency validation of the 3D data that is matched.

11. A method of directing a vehicle using a vision system, the method comprising the steps of:
producing a colorized 3D point cloud with data from a stereo visible light camera;
fusing data from a stereo long wave infrared (LWIR) camera with said 3D point cloud;
fusing data from a near infrared (NIR) camera with said 3D point cloud;
detecting foliage via fusing by way of measuring and comparing NIR and red spectrum energy level ratios,
filling in information via the LWIR camera data in areas of said 3D point cloud including areas where there is low light at night or dust which obscures a field of vision of said stereo visible light camera;
filling in regions, including depth regions via a segmentation computation and a hole-filling computation between sparse points to fuse with the 3D point cloud; and
producing from the fused, detected, and filled-in 3D data an enhanced 3D point cloud that is used to direct tasks of the vehicle.

12. The method of claim 11 , further comprising the step of directing a drive system of the vehicle to at least one of steer, change velocity, start and stop the vehicle, dependent upon said enhanced 3D point cloud.

13. The method of claim 11 , wherein said vision system is further configured to extract features from the 3D point cloud and the 3D data as part of producing said enhanced 3D point cloud.

14. The method of claim 13 , wherein said vision system is further configured to match at least some of the extracted features from the 3D point cloud and the 3D data as part of producing said enhanced 3D point cloud.

15. The method of claim 14 , wherein said vision system is further configured to perform a consistency validation of the 3D data that is matched.

Assignees

• Cnh Ind America Llc
• Autonomous Solutions Inc

Inventors

• Posselius John H
• Foster Christopher A
• Turpin Bret T
• Morwood Daniel J
• Petroff Thomas M
• Baillio Brad A
• Jeppesen Chad B

Classifications

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• H04N13/025

  Electricity · mapped topic

• G06T2207/10024

  Color image · CPC title

• G06K9/6202

  Physics · mapped topic

• G06K9/00791

  Physics · mapped topic