Close-in sensing camera system

Patent metadata

Field	Value
Publication number	US-11887378-B2
Application number	US-202218081260-A
Country	US
Kind code	B2
Filing date	Dec 14, 2022
Priority date	Dec 30, 2019
Publication date	Jan 30, 2024
Grant date	Jan 30, 2024

Abstract

The technology relates to an exterior sensor system for a vehicle configured to operate in an autonomous driving mode. The technology includes a close-in sensing (CIS) camera system to address blind spots around the vehicle. The CIS system is used to detect objects within a few meters of the vehicle. Based on object classification, the system is able to make real-time driving decisions. Classification is enhanced by employing cameras in conjunction with lidar sensors. The specific arrangement of multiple sensors in a single sensor housing is also important to object detection and classification. Thus, the positioning of the sensors and support components are selected to avoid occlusion and to otherwise prevent interference between the various sensor housing elements.

First claim

The invention claimed is:

1. An external sensing system for a vehicle configured to operate in an autonomous driving mode, the external sensing system comprising:
a lidar sensor arranged along an external sensing assembly, the lidar sensor having a lidar field of view configured to detect objects in a region of an external environment around the vehicle and within a threshold distance of the vehicle, the lidar field of view including an occlusion region within an immediate vicinity of the vehicle, the lidar field of view extending to at least the threshold distance of the vehicle;
an image sensor positioned relative to the lidar sensor along the external sensing assembly to have an image field of view that is within the region of the external environment and within the threshold distance of the vehicle, the image field of view at least partly overlapping with the lidar field of view and encompassing at least a portion of the occlusion region of the lidar field of view; and
a control system operatively coupled to the image sensor and the lidar sensor, the control system including one or more processors configured to:
receive lidar data from the lidar sensor and captured imagery from the image sensor;
perform processing of at least one of the lidar data or the captured imagery to detect an object in the region within the threshold distance of the vehicle;
classify the detected object; and
determine whether to cause one or more systems of the vehicle to perform a driving action based on classification of the detected object.

2. The external sensing system of claim 1 , wherein the image sensor is configured to provide a selected resolution for detection of objects within the threshold distance.

3. The external sensing system of claim 1 , wherein the image sensor is disposed no more than 0.3 meters from the lidar sensor.

4. The external sensing system of claim 1 , wherein the image sensor is aligned vertically below the lidar sensor when the external sensing assembly is disposed on the vehicle.

5. The external sensing system of claim 1 , wherein the image sensor is aligned vertically above the lidar sensor when the external sensing assembly is disposed on the vehicle.

6. The external sensing system of claim 1 , wherein the image sensor is further positioned relative to the lidar sensor to avoid occlusions between the image sensor and the lidar sensor.

7. The external sensing system of claim 1 , further comprising a separating surface disposed between the lidar sensor and the image sensor.

8. The external sensing system of claim 7 , wherein the separating surface is arranged having a downward sloping angle away from a surface of the external sensing assembly.

9. The external sensing system of claim 1 , wherein classification of the detected object includes determination of at least one of a size, proximity or orientation of the detected object.

10. The external sensing system of claim 1 , wherein the image sensor is configured to observe a minimum threshold volume taken up by the detected object.

11. The external sensing system of claim 1 , further comprising at least one illuminator configured to illuminate the image field of view of the image sensor, the at least one illuminator being distinct from the lidar sensor.

12. The external sensing system of claim 1 , further comprising at least one cleaning mechanism configured to clean the image sensor.

13. The external sensing system of claim 1 , further comprising a sensor conditioning element to reduce condensation or perform defrosting of at least one of the image sensor or the lidar sensor.

14. A vehicle configured to operate in an autonomous driving mode, the vehicle comprising:
a driving system including:
a deceleration system configured to control braking of the vehicle;
an acceleration system configured to control acceleration of the vehicle;
a steering system configured to control wheel orientation and a direction of the vehicle;
an external sensing system including:
a lidar sensor arranged along an external sensing assembly, the lidar sensor having a lidar field of view configured to detect objects in a region of an external environment around the vehicle and within a threshold distance of the vehicle, the lidar field of view including an occlusion region within an immediate vicinity of the vehicle, the lidar field of view extending to at least the threshold distance of the vehicle; and
an image sensor positioned relative to the lidar sensor along the external sensing assembly to have an image field of view that is within the region of the external environment and within the threshold distance of the vehicle, the image field of view at least partly overlapping with the lidar field of view and encompassing at least a portion of the occlusion region of the lidar field of view; and

a control system operatively coupled to the driving system and the external sensing system, the control system including one or more processors configured to:
receive lidar data from the lidar sensor and captured imagery from the image sensor;
perform processing of at least one of the lidar data or the captured imagery to detect an object in the region within the threshold distance of the vehicle;
classify the detected object; and
cause at least one of the driving system or the external sensing system to perform an action based on classification of the detected object.

15. The vehicle of claim 14 , wherein the action is a driving action by the driving system to modify a driving operation of the vehicle.

16. The vehicle of claim 14 , wherein the action is a sensor action to obtain additional lidar data or to capture additional imagery.

17. The vehicle of claim 14 , wherein the image sensor is aligned vertically below or above the lidar sensor.

18. The vehicle of claim 14 , wherein the image sensor is further positioned relative to the lidar sensor to avoid occlusions between the image sensor and the lidar sensor.

19. The vehicle of claim 14 , wherein classification of the detected object includes determination of at least one of a size, proximity or orientation of the detected object.

20. The vehicle of claim 14 , wherein the image sensor is configured to observe a minimum threshold volume taken up by the detected object.

21. The vehicle of claim 14 , further comprising at least one illuminator configured to illuminate the image field of view of the image sensor, the at least one illuminator being distinct from the lidar sensor.

22. The vehicle of claim 14 , further comprising at least one cleaning mechanism configured to clean the image sensor.

Assignees

• Waymo Llc

Inventors

• Toth Kimberly
• Dittmer Jeremy
• Guidi Giulia
• Avram Peter

Classifications

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• G06V20/58 Primary

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• B60R11/04 Primary

  Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle · CPC title

• G01S17/894

  Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar · CPC title

• G01S17/931

  of land vehicles · CPC title