Out of sequence perception system

Patent metadata

Field	Value
Publication number	US-12449543-B2
Application number	US-202418417356-A
Country	US
Kind code	B2
Filing date	Jan 19, 2024
Priority date	Aug 23, 2019
Publication date	Oct 21, 2025
Grant date	Oct 21, 2025

Abstract

Techniques for updating data operations in a perception system are discussed herein. A vehicle may use a perception system to capture data about an environment proximate to the vehicle. The perception system may receive image data, lidar data, and/or radar data to determine information about an object in the environment. As different sensors may be associated with different time periods for capturing and/or processing operations, the techniques include updating object data with data from sensors associated with a shorter time period to generate intermediate object data. Such intermediate object data may reduce a delay in updating a position of an object in an environment.

First claim

What is claimed is:

1. A method comprising:
receiving first sensor data captured by a first set of sensors of first sensor modalities, wherein the first set of sensors generate the first sensor data over a first time period;
generating a track for an object based at least in part on the first sensor data;
receiving second sensor data captured by a second sensor of a second sensor modality, wherein:
the second sensor generates the second sensor data over a second time period that is less than the first time period, and
the second sensor data is generated while third sensor data captured by the first set of sensors is unavailable due to the first set of sensors capturing the third sensor data over a third time period that is equal to the first time period;

one of updating the track or generating a new track for the object based solely on the second sensor data; and
controlling a vehicle based at least in part on at least one of the track or the new track.

2. The method of claim 1 , wherein the second sensor modality is radar.

3. The method of claim 1 , wherein generating the track comprises preprocessing the first sensor data, the preprocessing of the first sensor data includes at least one of detecting, classifying, or segmenting the first sensor data.

4. The method of claim 1 , wherein the first sensor modalities include lidar or cameras.

5. The method of claim 1 , wherein:
the track is associated with the object over a period of time including the first time period; the track or the new track is associated with the object over the period of time including the first time period and the second time period; and
the track and the new track comprise one or more of a position, an orientation, a velocity, or an acceleration.

6. The method of claim 1 , wherein the first sensor data comprises at least one of lidar data, image data, or radar data.

7. The method of claim 1 , further comprising one or more of receiving or determining initial object data associated with the object, wherein:
generating the track comprises preprocessing the first sensor data and performing association on the first sensor data with the initial object data.

8. The method of claim 1 , wherein the third time period at least partially overlaps with the second time period.

9. One or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform operations comprising:
receiving first sensor data captured by a first set of sensors of first sensor modalities, wherein the first set of sensors generate the first sensor data over a first time period;
generating a first track for an object based at least in part on the first sensor data;
receiving second sensor data captured by a second sensor of a radar sensor modality, wherein:
the second sensor generates the second sensor data over a second time period that is less than the first time period,
the second sensor data is generated while third sensor data captured over a third time period by the first set of sensors is unavailable, the third sensor data being unavailable due to the third time period being greater than the second time period; and
generating a second track based at least in part on the second sensor data; and

controlling a vehicle based at least in part on at least one of the first track or the second track.

10. The one or more non-transitory computer-readable media of claim 9 , wherein the first sensor modalities include radar.

11. The one or more non-transitory computer-readable media of claim 9 , wherein generating the first track comprises preprocessing the first sensor data, the preprocessing of the first sensor data includes at least one of detecting, classifying, or segmenting the first sensor data.

12. The one or more non-transitory computer-readable media of claim 9 , wherein the first sensor modalities include lidar and cameras.

13. The one or more non-transitory computer-readable media of claim 9 , wherein:
the first track is associated with the object over a period of time including the first time period;
the first track or the second track is associated with the object over the period of time including the first time period and the second time period; and
the first track and the second track comprise one or more of a position, an orientation, a velocity, or an acceleration.

14. The one or more non-transitory computer-readable media of claim 9 , wherein the first sensor data comprises at least one of lidar data, image data, or radar data.

15. The one or more non-transitory computer-readable media of claim 9 , the operations further comprising one or more of receiving or determining initial object data associated with the object, wherein:
generating the first track comprises preprocessing the first sensor data and performing association on the first sensor data with the initial object data.

16. The one or more non-transitory computer-readable media of claim 9 , wherein the second track is determined by updating the first track using exclusively the second sensor data.

17. A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions that, when executed, cause the one or more processors to perform operations comprising:
receiving first sensor data captured by a first set of sensors of first sensor modalities, wherein the first set of sensors generate the first sensor data over a first time period;
generating a track for an object based at least in part on the first sensor data;
receiving second sensor data captured by a second sensor of a radar sensor modality, wherein:
the second sensor generates the second sensor data over a second time period that is less than the first time period, and
the second sensor data is generated while third sensor data captured by the first set of sensors is unavailable due to the third sensor data being captured over a third time period that is greater than the first time period; based at least in part on the second sensor data, at least one of generating:
an updated track for the object, or a new track for the object; and

controlling a vehicle based at least in part on the updated track or the new track.

18. The system of claim 17 , wherein the first sensor modalities include radar.

19. The system of claim 17 , wherein:
the track is associated with the object over a period of time including the first time period;
the updated track or the new track is associated with the object over the period of time including the first time period and the second time period; and
the track, the updated track, and the new track comprise one or more of a position, an orientation, a velocity, or an acceleration.

20. The system of claim 17 , wherein the updated track or the new track is determined by updating the track using exclusively the second sensor data.

Assignees

• Zoox Inc

Inventors

• Das Subhasis
• Wang Chuang
• Pradhan Sabeek Mani

Classifications

• G01S17/66

  Tracking systems using electromagnetic waves other than radio waves · CPC title

• G01S7/4802

  using analysis of echo signal for target characterisation; Target signature; Target cross-section · CPC title

• G01S17/42

  Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46) · CPC title

• G01S17/58

  Velocity or trajectory determination systems; Sense-of-movement determination systems · CPC title

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title