Systems and methods for sensor data packet processing and spatial memory updating for robotic platforms

Patent metadata

Field	Value
Publication number	US-12350835-B2
Application number	US-202418670288-A
Country	US
Kind code	B2
Filing date	May 21, 2024
Priority date	Jul 29, 2020
Publication date	Jul 8, 2025
Grant date	Jul 8, 2025

Abstract

Systems and methods for streaming sensor packets in real-time are provided. An example method includes obtaining a sensor data packet representing a first portion of a three-hundred and sixty degree view of a surrounding environment of a robotic platform. The method includes generating, using machine-learned model(s), a local feature map based at least in part on the sensor data packet. The local feature map is indicative of local feature(s) associated with the first portion of the three-hundred and sixty degree view. The method includes updating, based at least in part on the local feature map, a spatial map to include the local feature(s). The spatial map includes previously extracted local features associated with a previous sensor data packet representing a different portion of the three-hundred and sixty degree view than the first portion. The method includes determining an object within the surrounding environment based on the updated spatial map.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
obtaining, from a sensor system, a stream of sensor data packets representing respective angular slices of a view of a surrounding environment of an autonomous vehicle, wherein the stream of sensor data packets includes a first sensor data packet corresponding to a first angular slice of the view that is obtained at a first time when the first sensor data packet is acquired through the sensor system;
generating a first local feature map based at least in part on the first sensor data packet, wherein the first local feature map is indicative of one or more first local features associated with the first angular slice of the view;
updating a spatial map to include the first local features, wherein the spatial map comprises previously extracted local features associated with a previous sensor data packet representing a different angular slice of the view of the surrounding environment than the first angular slice;
determining an object within the surrounding environment of the autonomous vehicle based at least in part on the spatial map including the first local features; and
controlling motion of the autonomous vehicle based at least in part on the object determined within the surrounding environment.

2. The computer-implemented method of claim 1 , wherein the stream of sensor data packets are received from the sensor system and iteratively processed in a streaming fashion as respective angular slices corresponding to a three-hundred and sixty degree view are obtained and updated in the spatial map.

3. The computer-implemented method of claim 1 , wherein the sensor system comprises one or more LIDAR sensors.

4. The computer-implemented method of claim 3 , wherein the sensor system ingests packets of LIDAR data from the one or more LIDAR sensors and emits the stream of sensor data packets without waiting for a full sweep to be built by the one or more LIDAR sensors.

5. The computer-implemented method of claim 3 , wherein the first sensor data packet comprises three-dimensional point cloud data generated by the one or more LIDAR sensors, wherein the one or more LIDAR sensors are rotated to generate sensor data packets for each respective angular slice of the view.

6. The computer-implemented method of claim 1 , wherein the stream of sensor data packets further includes a second sensor data packet corresponding to a second angular slice of the view that is obtained at a second time when the second sensor data packet is acquired through the sensor system after the first sensor data packet.

7. The computer-implemented method of claim 6 , further comprising:
generating a second local feature map based at least in part on the second sensor data packet, wherein the second local feature map is indicative of one or more second local features associated with the second angular slice of the view of the surrounding environment; and
updating, based at least in part on the second local feature map, the spatial map to include the one or more second local features;
wherein the second local feature map is generated after the first local feature map is generated.

8. The computer-implemented method of claim 1 , wherein generating the first local feature map comprises generating a two-dimensional representation associated with the first angular slice of the view of the surrounding environment of the autonomous vehicle.

9. The computer-implemented method of claim 1 , wherein determining the object comprises obtaining map data and fusing the map data with the spatial map.

10. The computer-implemented method of claim 1 , wherein updating the spatial map comprises replacing one or more previous computations of the spatial map with the one or more first local features.

11. The computer-implemented method of claim 1 , wherein updating the spatial map comprises:
obtaining previous computations descriptive of past computations for previous sensor data packets; and
updating spatial memory based on the previous sensor data packets and the first local feature map to update the spatial map.

12. The computer-implemented method of claim 1 , wherein the spatial map is descriptive of past local feature maps generated by processing previously obtained sensor data packets.

13. A computing system comprising:
one or more processors; and
one or more computer-readable medium storing instructions for execution by the one or more processors to cause the computing system to perform operations, the operations comprising:
obtaining, from a sensor system, a stream of sensor data packets representing respective angular slices of a view of a surrounding environment of an autonomous vehicle, wherein the stream of sensor data packets includes a first sensor data packet corresponding to a first angular slice of the view that is obtained at a first time when the first sensor data packet is acquired through the sensor system;
generating a first local feature map based at least in part on the first sensor data packet, wherein the first local feature map is indicative of one or more first local features associated with the first angular slice of the view;
updating a spatial map to include the first local features, wherein the spatial map comprises previously extracted local features associated with a previous sensor data packet representing a different angular slice of the view of the surrounding environment than the first angular slice;
determining an object within the surrounding environment of the autonomous vehicle based at least in part on the spatial map including the first local features; and
controlling motion of the autonomous vehicle based at least in part on the object determined within the surrounding environment.

14. The computing system of claim 13 , wherein the stream of sensor data packets are received from the sensor system and iteratively processed in a streaming fashion as respective angular slices corresponding to a three-hundred and sixty degree view are obtained and updated in the spatial map.

15. The computing system of claim 13 , wherein:
the sensor system comprises one or more LIDAR sensors; and
the sensor system ingests packets of LIDAR data from the one or more LIDAR sensors and emits the stream of sensor data packets without waiting for a full sweep to be built by the one or more LIDAR sensors.

16. The computing system of claim 15 , wherein the first sensor data packet comprises three-dimensional point cloud data generated by the one or more LIDAR sensors, wherein the one or more LIDAR sensors are rotated to generate sensor data packets for each respective angular slice of the view.

17. The computing system of claim 13 , wherein the stream of sensor data packets further includes a second sensor data packet corresponding to a second angular slice of the view that is obtained at a second time when the second sensor data packet is acquired through the sensor system after the first sensor data packet.

18. The computing system of claim 17 , the operations further comprising:
generating a second local feature map based at least in part on the second sensor data packet, wherein the second local feature map is indicative of one or more second local features associated with the second angular slice of the view of the surrounding environment; and

updating, based at least in part on the second local feature map, the spatial map to include the one or more second local features;
wherein the second local feature map is generated after the first local feature map is generated.

19. An autonomo

Assignees

• Aurora Operations Inc

Inventors

• Casas Sergio
• Frossard Davi Eugenio Nascimento
• Suo Shun Da
• Tu Xuanyuan
• Urtasun Raquel

Classifications

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06F18/253

  of extracted features · CPC title

• B25J9/1664

  characterised by motion, path, trajectory planning · CPC title