Off-board perception based on sequence to sequence sensor data generation

Patent metadata

Field	Value
Publication number	US-12565237-B2
Application number	US-202418901743-A
Country	US
Kind code	B2
Filing date	Sep 30, 2024
Priority date	Oct 4, 2023
Publication date	Mar 3, 2026
Grant date	Mar 3, 2026

Abstract

Methods, systems, and non-transitory computer readable media are configured to perform operations comprising capturing a first sequence of captured data associated with a first time window; generating a second sequence of generated data associated with a second time window based on the first sequence of data; identifying a difference between the second sequence of generated data and a ground truth sequence of captured data associated with the second time window; determining whether the difference between the second sequence of generated data and the ground truth sequence of captured data satisfies a selected threshold value; detecting a change associated with an environment when the difference between the second sequence of generated data and the ground truth sequence of captured data satisfies the selected threshold value; and based on the detected change, providing navigation guidance to a vehicle travelling in the environment.

First claim

What is claimed is:

1 . A computer-implemented method comprising:
capturing, by a computing system, a first sequence of captured data associated with a first time window, wherein the first sequence of captured data includes image data; generating, by the computing system, a second sequence of generated data associated with a second time window based on the first sequence of data, wherein the generating is performed by an infrastructure pod associated with a segment of an environment; identifying, by the computing system, a difference between the second sequence of generated data and a ground truth sequence of captured data associated with the second time window, wherein the difference between the second sequence of generated data and the ground truth sequence of captured data is associated with pixel data; determining, by the computing system, whether the difference between the second sequence of generated data and the ground truth sequence of captured data satisfies a selected threshold value; detecting, by the computing system, a change associated with an environment when the difference between the second sequence of generated data and the ground truth sequence of captured data satisfies the selected threshold value, the detected change relating to appearance of an object in the environment; and based on the detected change, providing, by the computing system, navigation guidance to a vehicle travelling in the environment.

2 . The computer-implemented method of claim 1 , wherein the ground truth sequence of captured data associated with the second time window follows the first sequence of captured data associated with the first time window in a sequence of sensor data capturing a segment of an environment for which an infrastructure system provides services.

3 . The computer-implemented method of claim 1 ,
wherein the navigation guidance comprises at least one of i) a location of the detected change and ii) a suggestion to perform a navigation maneuver in response to the detected change.

4 . The computer-implemented method of claim 1 , wherein the detected change is an object or event associated with a frequency of appearance that is less than a selected threshold frequency value.

5 . The computer-implemented method of claim 1 , wherein the generating is performed by a sequence to sequence neural network.

6 . The computer-implemented method of claim 5 , further comprising:
training, by the computing system, the sequence to sequence neural network based on training data to remove an object or event from sequences of generated data outputted by the sequence to sequence neural network.

7 . The computer-implemented method of claim 1 , wherein the identifying is based on a difference map generated by a machine learning model that reflects differences associated with frame pairs from two sequences of data.

8 . The computer-implemented method of claim 1 , wherein the first sequence of captured data associated with the first window of time is a first number of frames of sensor data captured by an infrastructure system providing service to vehicles travelling in an environment, and the second sequence of generated data and the ground truth sequence of captured data are a second number of frames that is different from the first number of frames.

9 . A system comprising:
at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising: capturing a first sequence of captured data associated with a first time window, wherein the first sequence of captured data includes image data; generating a second sequence of generated data associated with a second time window based on the first sequence of data, wherein the generating is performed by an infrastructure pod associated with a segment of an environment; identifying a difference between the second sequence of generated data and a ground truth sequence of captured data associated with the second time window, wherein the difference between the second sequence of generated data and the ground truth sequence of captured data is associated with pixel data; determining whether the difference between the second sequence of generated data and the ground truth sequence of captured data satisfies a selected threshold value; detecting a change associated with an environment when the difference between the second sequence of generated data and the ground truth sequence of captured data satisfies the selected threshold value, the detected change relating to appearance of an object in the environment; and based on the detected change, providing navigation guidance to a vehicle travelling in the environment.

10 . The system of claim 9 , wherein the ground truth sequence of captured data associated with the second time window follows the first sequence of captured data associated with the first time window in a sequence of sensor data capturing a segment of an environment for which an infrastructure system provides services.

11 . The system of claim 9 , wherein
the navigation guidance comprises at least one of i) a location of the detected change and ii) a suggestion to perform a navigation maneuver in response to the detected change.

12 . The system of claim 9 , wherein the detected change is an object or event associated with a frequency of appearance that is less than a selected threshold frequency value.

13 . A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform operations comprising:
capturing a first sequence of captured data associated with a first time window, wherein the first sequence of captured data includes image data; generating a second sequence of generated data associated with a second time window based on the first sequence of data, wherein the generating is performed by an infrastructure pod associated with a segment of an environment; identifying a difference between the second sequence of generated data and a ground truth sequence of captured data associated with the second time window, wherein the difference between the second sequence of generated data and the ground truth sequence of captured data is associated with pixel data; determining whether the difference between the second sequence of generated data and the ground truth sequence of captured data satisfies a selected threshold value; detecting a change associated with an environment when the difference between the second sequence of generated data and the ground truth sequence of captured data satisfies the selected threshold value, the detected change relating to appearance of an object in the environment; and based on the detected change, providing navigation guidance to a vehicle travelling in the environment.

14 . The non-transitory computer-readable storage medium of claim 13 , wherein the ground truth sequence of captured data associated with the second time window follows the first sequence of captured data associated with the first time window in a sequence of sensor data capturing a segment of an environment for which an infrastructure system provides services.

15 . The non-transitory computer-readable storage medium of claim 13 , wherein the operations further comprise:
based on the detected change, providing navigation guidance to a vehicle travelling in the environment, the navigation guidance comprising at least one of i) a location of the detected change and ii) a suggestion to perform a navigation maneuver in response to the detected

Assignees

• Plusai Inc

Inventors

• Ganguli Anurag
• Daly Jr Timothy Patrick
• Zheng Hao

Classifications

• G06N3/08 Primary

  Learning methods · CPC title

• G01C21/30

  Map- or contour-matching · CPC title

• B60W2554/4044

  Direction of movement, e.g. backwards · CPC title

• B60W2420/408

  Radar; Laser, e.g. lidar · CPC title

• B60W2554/4042

  Longitudinal speed · CPC title