Method and system for dynamically updating an environmental representation of an autonomous agent

Patent metadata

Field	Value
Publication number	US-12371067-B2
Application number	US-202217949921-A
Country	US
Kind code	B2
Filing date	Sep 21, 2022
Priority date	Dec 17, 2020
Publication date	Jul 29, 2025
Grant date	Jul 29, 2025

Abstract

The method for dynamically updating an environmental representation of an autonomous agent can include: receiving a set of inputs S210; generating an environmental representation S220; and updating the environmental representation S230. Additionally or alternatively, the method S200 can include providing the environmental representation to a planning module S240 and/or any other suitable processes. The method S200 functions to generate and/or dynamically update an environmental representation to facilitate control of an autonomous agent.

First claim

We claim:

1. A method for operating an autonomous vehicle, the method comprising:
collecting a first set of data from a set of sensors associated with the autonomous vehicle;
based on the first set of data, generating a first environmental representation of the autonomous vehicle's environment at a first time point, and wherein the first environmental representation comprises a first set of labeled objects;
collecting a second set of data from the set of sensors;
based on the second set of data, generating a second environmental representation of the autonomous vehicle's environment at a second time point, the second time point occurring later in the time than the first time point, and wherein the second environmental representation comprises a second set of labeled objects;
evaluating the second set of labeled objects according to a set of hypotheses;
based on evaluation of the second set of labeled objects, relabeling the second set of labeled objects in the second environmental representation to produce a refined second environmental representation;
retroactively relabeling the first set of labeled objects in the first environmental representation to produce a refined first environmental representation; and
operating the autonomous vehicle based on at least one of the refined first environmental representation and the refined second environmental representation.

2. The method of claim 1 , wherein the set of hypotheses comprises:
a first subset of hypotheses, the first subset of hypotheses comprising object merge hypotheses; and
a second subset of hypotheses, the second subset of hypotheses comprising object split hypotheses.

3. The method of claim 2 , wherein each of the second set of labeled objects is evaluated based on a hypothesis of the first subset and a hypothesis of the second subset.

4. The method of claim 1 , wherein evaluating the second set of labeled objects according to the set of hypotheses comprises determining a score for each of the set of hypotheses to produce a set of scores, wherein labels of the relabeled second set of objects are determined based on the set of scores.

5. The method of claim 4 , wherein the set of scores is determined based on a set of multiple historical environmental representations.

6. The method of claim 5 , wherein determining the set of scores based on the set of historical environmental representations comprises evaluating a smoothness of trajectories of the labeled second set of objects based on the set of multiple historical environmental representations.

7. The method of claim 5 , wherein the set of historical environmental representations comprises the first environmental representation.

8. The method of claim 1 , further comprising operating the autonomous vehicle based on both the refined first environmental representation and the refined second environmental representation.

9. The method of claim 1 , wherein retroactively relabeling the first set of labeled objects is performed in response to relabeling the second set of labeled objects.

10. A system for operating an autonomous vehicle, the system comprising:
a memory comprising Random Access Memory (RAM) storing a set of historical environmental representations; and
a processing subsystem comprising a Central Processing Unit (CPU) in communication with the memory and a drive-by-wire system of the autonomous vehicle, the processing subsystem configured to:
collect a first set of data from a set of sensors associated with the autonomous vehicle;
based on the first set of data, generate a first environmental representation of the autonomous vehicle's environment at a first time point, and wherein the first environmental representation comprises a first set of labeled objects;
store the first environmental representation in the set of historical environmental representations of the memory;
collect a second set of data from the set of sensors;
based on the second set of data, generate a second environmental representation of the autonomous vehicle's environment at a second time point, the second time point occurring later in the time than the first time point, and wherein the second environmental representation comprises a second set of labeled objects;
evaluate the second set of labeled objects according to a set of hypotheses;
based on evaluation of the second set of labeled objects, relabel the second set of labeled objects in the second environmental representation to produce a refined second environmental representation;
retroactively relabel the first set of labeled objects in the first environmental representation to produce a refined first environmental representation; and
control the drive-by-wire system of the autonomous vehicle based on at least one of the refined first environmental representation and the refined second environmental representation.

11. The system of claim 10 , further comprising the drive-by-wire system of the autonomous vehicle, wherein the drive-by-wire system comprises a controller, wherein controlling the drive-by-wire system comprises steering the autonomous vehicle based on instructions from the CPU.

12. The system of claim 10 , wherein the processing subsystem is further configured to replace the first environmental representation with the refined first environmental representation in the set of historical environmental representations in the memory.

13. The system of claim 10 , wherein the set of hypotheses comprises:
a first subset of hypotheses, the first subset of hypotheses comprising object merge hypotheses; and
a second subset of hypotheses, the second subset of hypotheses comprising object split hypotheses.

14. The system of claim 13 , wherein each of the second set of labeled objects is evaluated based on a hypothesis of the first subset and a hypothesis of the second subset.

15. The system of claim 13 , wherein evaluating the set of hypotheses comprises evaluating a set of distances between objects of the second set of labeled objects.

16. The system of claim 10 , wherein evaluating the second set of labeled objects according to the set of hypotheses comprises determining a score for each of the set of hypotheses to produce a set of scores, wherein labels of the relabeled second set of objects are determined based on the set of scores.

17. The system of claim 16 , wherein the set of scores is determined based on the set of historical environmental representations, the set of historical environmental representations comprising multiple historical environmental representations.

18. The system of claim 17 , wherein determining the set of scores based on the set of historical environmental representations comprises evaluating a smoothness of trajectories of the labeled second set of objects based on the set of historical environmental representations.

19. The system of claim 10 , wherein the autonomous vehicle is operated based on both the refined first environmental representation and the refined second environmental representation.

20. The system of claim 10 , wherein retroactively relabeling the first set of labeled objects is performed in response to relabeling the second set of labeled objects.

Assignees

• May Mobility Inc

Inventors

• Goeddel Robert
• Johnson Collin
• Kothbauer Melinda
• Sterniak Jeff
• Voorheis Tom
• Olson Edwin B

Classifications

• B60W2420/403

  Image sensing, e.g. optical camera · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• B60W2556/10

  Historical data · CPC title

• G06T2207/30241

  Trajectory · CPC title

• G06V20/58 Primary

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title