Open vehicle doors prediction using a neural network model

Patent metadata

Field	Value
Publication number	US-12497074-B2
Application number	US-202217940665-A
Country	US
Kind code	B2
Filing date	Sep 8, 2022
Priority date	Sep 9, 2021
Publication date	Dec 16, 2025
Grant date	Dec 16, 2025

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for open vehicle doors prediction using a neural network model. One of the methods includes: obtaining sensor data (i) that includes a portion of a point cloud generated by a laser sensor of an autonomous vehicle and (ii) that characterizes a vehicle that is in a vicinity of the autonomous vehicle in an environment; and processing the sensor data using an open door prediction neural network to generate an open door prediction that predicts a likelihood score that the vehicle has an open door.

First claim

What is claimed is:

1 . A method performed by one or more computers, the method comprising:
obtaining sensor data (i) that comprises a portion of a point cloud generated by a laser sensor of an autonomous vehicle and (ii) that characterizes a second vehicle that is in a vicinity of the autonomous vehicle in an environment, wherein the sensor data further comprises an image patch depicting the second vehicle generated from an image of the environment captured by a camera sensor; processing the sensor data using an open door prediction neural network to generate an open door prediction that predicts a likelihood score that the second vehicle has an open door, wherein the open door prediction neural network comprises:
a first embedding subnetwork that is configured to process the portion of the point cloud to generate a point cloud embedding characterizing the second vehicle;
a second embedding subnetwork that is configured to process the image patch to generate an image embedding characterizing the second vehicle; and
an output subnetwork that is configured to process the point cloud embedding and the image embedding to generate the likelihood score that the second vehicle has an open door; and
providing an input comprising the open door prediction to a planning system of the autonomous vehicle to plan a future trajectory of the autonomous vehicle.

2 . The method of claim 1 , wherein the open door prediction further comprises an open door segmentation prediction that comprises, for each point of the portion of the point cloud, a per-point likelihood that predicts a likelihood that the point corresponds to the open door.

3 . The method of claim 2 , further comprising:
providing an input comprising the open door segmentation prediction to a vehicle segmentation model that is used by the planning system of the autonomous vehicle to generate a segmentation prediction of the second vehicle.

4 . The method of claim 1 , wherein the open door prediction neural network comprises a PointNet neural network.

5 . The method of claim 1 , wherein the open door prediction neural network comprises a Range Sparse Net neural network.

6 . The method of claim 1 , wherein the sensor data further comprises a second portion of a second point cloud generated by a second laser sensor that has a shorter range than the laser sensor.

7 . The method of claim 1 , wherein the open door prediction neural network comprises:
a concatenation layer that concatenates the point cloud embedding and the image embedding to generate a concatenated embedding; and the output subnetwork is configured to process the concatenated embedding to generate the likelihood score that the second vehicle has an open door.

8 . The method of claim 1 , wherein the open door comprises at least one of the following: a left door, a right door, a hood, a trunk, a sliding door.

9 . A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
obtaining sensor data (i) that comprises a portion of a point cloud generated by a laser sensor of an autonomous vehicle and (ii) that characterizes a second vehicle that is in a vicinity of the autonomous vehicle in an environment, wherein the sensor data further comprises an image patch depicting the second vehicle generated from an image of the environment captured by a camera sensor; processing the sensor data using an open door prediction neural network to generate an open door prediction that predicts a likelihood score that the second vehicle has an open door, wherein the open door prediction neural network comprises:
a first embedding subnetwork that is configured to process the portion of the point cloud to generate a point cloud embedding characterizing the second vehicle;
a second embedding subnetwork that is configured to process the image patch to generate an image embedding characterizing the second vehicle; and
an output subnetwork that is configured to process the point cloud embedding and the image embedding to generate the likelihood score that the second vehicle has an open door; and
providing an input comprising the open door prediction to a planning system of the autonomous vehicle to plan a future trajectory of the autonomous vehicle.

10 . The system of claim 9 , wherein the open door prediction further comprises an open door segmentation prediction that comprises, for each point of the portion of the point cloud, a per-point likelihood that predicts a likelihood that the point corresponds to the open door.

11 . The system of claim 10 , the operations further comprise:
providing an input comprising the open door segmentation prediction to a vehicle segmentation model that is used by the planning system of the autonomous vehicle to generate a segmentation prediction of the second vehicle.

12 . The system of claim 9 , wherein the open door prediction neural network comprises a PointNet neural network.

13 . The system of claim 9 , wherein the open door prediction neural network comprises a Range Sparse Net neural network.

14 . The system of claim 9 , wherein the sensor data further comprises a second portion of a second point cloud generated by a second laser sensor that has a shorter range than the laser sensor.

15 . The system of claim 9 , wherein the open door prediction neural network comprises:
a concatenation layer that concatenates the point cloud embedding and the image embedding to generate a concatenated embedding; and the output subnetwork is configured to process the concatenated embedding to generate the likelihood score that the second vehicle has an open door.

16 . The system of claim 9 , wherein the open door comprises at least one of the following: a left door, a right door, a hood, a trunk, a sliding door.

17 . One or more non-transitory computer storage media encoded with computer program instructions that when executed by a plurality of computers cause the plurality of computers to perform operations comprising:
obtaining sensor data (i) that comprises a portion of a point cloud generated by a laser sensor of an autonomous vehicle and (ii) that characterizes a second vehicle that is in a vicinity of the autonomous vehicle in an environment, wherein the sensor data further comprises an image patch depicting the second vehicle generated from an image of the environment captured by a camera sensor; processing the sensor data using an open door prediction neural network to generate an open door prediction that predicts a likelihood score that the second vehicle has an open door, wherein the open door prediction neural network comprises:
a first embedding subnetwork that is configured to process the portion of the point cloud to generate a point cloud embedding characterizing the second vehicle;
a second embedding subnetwork that is configured to process the image patch to generate an image embedding characterizing the second vehicle; and
an output subnetwork that is configured to process the point cloud embedding and the image embedding to generate the likelihood score that the second vehicle has an open door; and
providing an input comprising the open door prediction to a planning system of the autonomous vehicle to plan a future trajectory of the autonomous vehicle.

18 . The non-transitory computer storage media of claim 17 , wherein the open door prediction further comprises an open door segmentation prediction that compri

Assignees

• Waymo Llc

Inventors

• Cornman Andre Liang
• Lee David
• Song Yang
• Guo Zijian
• Walker Jr Edward Stephen
• Wang Yuanfang
• Zhao Zhengli
• Chiu Hsu-Kuang

Classifications

• B60W2420/408

  Radar; Laser, e.g. lidar · CPC title

• B60W2420/403

  Image sensing, e.g. optical camera · CPC title

• B60W2554/4049

  Relationship among other objects, e.g. converging dynamic objects · CPC title

• G06V20/58 Primary

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06V10/82

  using neural networks · CPC title