Object bounding contours based on image data

Patent metadata

Field	Value
Publication number	US-12475718-B2
Application number	US-202418750796-A
Country	US
Kind code	B2
Filing date	Jun 21, 2024
Priority date	Oct 28, 2021
Publication date	Nov 18, 2025
Grant date	Nov 18, 2025

Abstract

Techniques are discussed herein for controlling autonomous vehicles within a driving environment, including generating and using bounding contours associated with objects detected in the environment. Image data may be captured and analyzed to identify and/or classify objects within the environment. Image-based and/or lidar-based techniques may be used to determine depth data associated with the objects, and a bounding contour may be determined based on the object boundaries and associated depth data. An autonomous vehicle may use the bounding contours of objects within the environment to classify the objects, predict the positions, poses, and trajectories of the objects, and determine trajectories and perform other vehicle control actions while safely navigating the environment.

First claim

What is claimed is:

1 . A method comprising:
receiving sensor data from a sensor associated with a vehicle in an environment; determining, based on the sensor data, object data representing an object in the environment; inputting the object data to a machine learning model; receiving, from the machine learning model, depth data associated with the object; determining a boundary edge associated with the object, based at least in part on at least one of the sensor data or the depth data; determining, based at least in part on the depth data and the boundary edge, a bounding contour associated with the object, the bounding contour including a first segment and a second segment connected by an interior point in an interior of the boundary edge; and controlling the vehicle within the environment based at least in part on the bounding contour associated with the object.

2 . The method of claim 1 , wherein the boundary edge has a first endpoint and a second endpoint, and determining the bounding contour associated with the object comprises:
replacing the boundary edge with:
the first segment based on the first endpoint and the interior point; and
the second segment based on the interior point and the second endpoint,
wherein the first segment and the second segment are based at least in part on the depth data.

3 . The method of claim 1 , wherein the depth data is first depth data, the method further comprising:
receiving second depth data from a depth sensor; and projecting the second depth data onto a two-dimensional plane associated with the sensor data, wherein determining the first depth data is based at least in part on the second depth data.

4 . The method of claim 3 , wherein determining the first depth data comprises:
determining a boundary region associated with the object, based at least in part on the sensor data; and determining the first depth data within the boundary region, based at least in part on the second depth data.

5 . The method of claim 1 , further comprising:
determining an object type associated with the object, based at least in part on the sensor data; and determining a depth model associated with the object type, wherein determining the bounding contour is based at least in part on the depth model.

6 . The method of claim 1 , wherein the bounding contour is a first bounding contour, the method further comprising:
receiving lidar data from a lidar sensor; and determining, based at least in part on the lidar data, a second bounding contour associated with the object, wherein controlling the vehicle is based at least in part on the first bounding contour and the second bounding contour.

7 . The method of claim 1 , wherein controlling the vehicle comprises:
determining a proposed trajectory for the vehicle within the environment; determining a point on the bounding contour, based at least in part on a depth model associated with the object, wherein the point is associated with a first surface of the object that is obscured within the sensor data; determining a predicted distance at a future time, between a portion of the vehicle and the point on the bounding contour, based at least in part on the proposed trajectory; and validating the proposed trajectory based at least in part on the predicted distance.

8 . The method of claim 1 , further comprising:
determining that the object is associated with a dynamic object type, based at least in part on an image-based object classification; and determining a trajectory associated with the object, based at least in part on the dynamic object type, wherein controlling the vehicle is based at least in part on the trajectory associated with the object.

9 . One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving sensor data from a sensor associated with a vehicle in an environment; determining, based on the sensor data, object data representing an object in the environment; inputting the object data to a machine learning model; receiving, from the machine learning model, depth data associated with the object; determining a boundary edge associated with the object, based at least in part on at least one of the sensor data or the depth data; determining, based at least in part on the depth data and the boundary edge, a bounding contour associated with the object, the bounding contour including a first segment and a second segment connected by an interior point in an interior of the boundary edge; and controlling the vehicle within the environment based at least in part on the bounding contour associated with the object.

10 . The one or more non-transitory computer-readable media of claim 9 , wherein the boundary edge has a first endpoint and a second endpoint, and determining the bounding contour associated with the object comprises:
replacing the boundary edge with:
the first segment based on the first endpoint and the interior point; and
the second segment based on the interior point and the second endpoint,
wherein the first segment and the second segment are based at least in part on the depth data.

11 . The one or more non-transitory computer-readable media of claim 9 , wherein the depth data is first depth data, the operations further comprising:
receiving second depth data from a depth sensor; and projecting the second depth data onto a two-dimensional plane associated with the sensor data, wherein determining the first depth data is based at least in part on the second depth data.

12 . The one or more non-transitory computer-readable media of claim 11 , wherein determining the first depth data comprises:
determining a boundary region associated with the object, based at least in part on the sensor data; and determining the first depth data within the boundary region, based at least in part on the second depth data.

13 . The one or more non-transitory computer-readable media of claim 9 , the operations further comprising:
determining an object type associated with the object, based at least in part on the sensor data; and determining a depth model associated with the object type, wherein determining the bounding contour is based at least in part on the depth model.

14 . The one or more non-transitory computer-readable media of claim 9 , wherein the bounding contour is a first bounding contour, the operations further comprising:
receiving lidar data from a lidar sensor; and determining, based at least in part on the lidar data, a second bounding contour associated with the object, wherein controlling the vehicle is based at least in part on the first bounding contour and the second bounding contour.

15 . The one or more non-transitory computer-readable media of claim 9 , wherein controlling the vehicle comprises:
determining a proposed trajectory for the vehicle within the environment; determining a point on the bounding contour, based at least in part on a depth model associated with the object, wherein the point is associated with a first surface of the object that is obscured within the sensor data; determining a predicted distance at a future time, between a portion of the vehicle and the point on the bounding contour, based at least in part on the proposed trajectory; and validating the proposed trajectory based at least in part on the predicted distance.

16 . A system comprising:
one or more processors; and one or more non-transitory computer-readabl

Assignees

• Zoox Inc

Inventors

• Purdy Scott M
• Ma Derek Xiang
• Das Subhasis
• Wang Zeng

Classifications

• B60W2420/408

  Radar; Laser, e.g. lidar · CPC title

• B60W2420/403

  Image sensing, e.g. optical camera · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• B60W60/0027

  using trajectory prediction for other traffic participants · CPC title

• G06T2207/30261

  Obstacle · CPC title