Systems and methods for augmenting perception data with supplemental information

Patent metadata

Field	Value
Publication number	US-11875680-B2
Application number	US-202016983414-A
Country	US
Kind code	B2
Filing date	Aug 3, 2020
Priority date	Mar 31, 2020
Publication date	Jan 16, 2024
Grant date	Jan 16, 2024

Abstract

Examples disclosed herein may involve a computing system that is configured to(i) obtain previously-derived perception data for a collection of sensor data including a sequence of frames observed by a vehicle within one or more scenes, where the previously-derived perception data includes a respective set of object-level information for each of a plurality of objects detected within the sequence of frames, (ii) derive supplemental object-level information for at least one object detected within the sequence of frames that adds to the previously-derived object-level information for the at least one object, (iii) augment the previously-derived perception data to include the supplemental object-level information for the at least one object, and (iv) store the augmented perception data in an arrangement that encodes a hierarchical relationship between the plurality of objects, the sequence of frames, and the one or more scenes.

First claim

We claim:

1. A computer-implemented method comprising:
maintaining a database of augmented perception data for scenes observed by one or more vehicles, wherein the augmented perception data comprises (a) object-level information for one or more objects detected within the scenes and (b) supplemental information for the scenes that is derived from, and adds to, the object-level information for the one or more objects;
executing a query of the database to identify an occurrence of a type of perception event within the scenes, wherein the query is based at least in part on the supplemental information included in the augmented perception data; and
based on the identified occurrence of the type of perception event, deriving a performance metric for an autonomy system that is indicative of perception performance for the autonomy system in scenes including the occurrence of the type of perception event.

2. The computer-implemented method of claim 1 , wherein the type of perception event comprises one of (i) a detection of a phantom object, (ii) an object class label switch, or (iii) an object tracking discontinuity.

3. The computer-implemented method of claim 1 , wherein the performance metric is for determining a capability of a specific version of the autonomy system to perform in one or more situations including the type of perception event.

4. The computer-implemented method of claim 1 , wherein the type of perception event is an object class label switch, and wherein the query of the database to identify the occurrence of the type of perception event comprises a query to identify instances where an object had a class label switch from a first value to a second value and then back to the first value again within a given number of sequential frames.

5. The computer-implemented method of claim 1 , wherein the query of the database to identify the occurrence of the type of perception event comprises a query of scenes observed by vehicles running a given version of a perception system.

6. The computer-implemented method of claim 1 , wherein the supplemental information for the scenes comprises supplemental geometric information that indicates a spatial relationship between at least one object and one of the one or more vehicles, and wherein the query of the database is based at least in part on the supplemental geometric information.

7. The computer-implemented method of claim 1 , wherein the augmented perception data in the database is derived by:
obtaining previously-derived perception data for the scenes observed by the one or more vehicles, wherein the previously-derived perception data comprises the object-level information for the one or more objects;
deriving the supplemental information for the scenes; and
augmenting the previously-derived perception data to include the supplemental information for the scenes.

8. The computer-implemented method of claim 1 , wherein the augmented perception data comprises supplemental scene-level information for at least one scene observed by the one or more vehicles that includes one or both of spatial information or temporal information, and wherein the query is based at least in part on the supplemental scene-level information.

9. The computer-implemented method of claim 1 , wherein the one or more objects were detected within respective sequences of frames observed by respective vehicles within the scenes, and wherein maintaining the database of augmented perception data comprises storing the augmented perception data in an arrangement that encodes a hierarchical relationship between the one or more objects, the respective sequences of frames, and the scenes.

10. A non-transitory computer-readable medium comprising program instructions stored thereon that are executable to cause a computing system to:
maintain a database of augmented perception data for scenes observed by one or more vehicles, wherein the augmented perception data comprises (a) object-level information for one or more objects detected within the scenes and (b) supplemental information for the scenes that is derived from, and adds to, the object-level information for the one or more objects;
execute a query of the database to identify an occurrence of a type of perception event within the scenes, wherein the query is based at least in part on the supplemental information included in the augmented perception data; and
based on the identified occurrence of the type of perception event, derive a performance metric for an autonomy system that is indicative of perception performance for the autonomy system in scenes including the occurrence of the type of perception event.

11. The computer-readable medium of claim 10 , wherein the type of perception event comprises one of (i) a detection of a phantom object, (ii) an object class label switch, or (iii) an object tracking discontinuity.

12. The computer-readable medium of claim 10 , wherein the performance metric is for determining a capability of a specific version of the autonomy system to perform in one or more situations including the type of perception event.

13. The computer-readable medium of claim 10 , wherein the type of perception event is an object class label switch, and wherein the query of the database to identify the occurrence of the type of perception event comprises a query to identify instances where an object had a class label switch from a first value to a second value and then back to the first value again within a given number of sequential frames.

14. The computer-readable medium of claim 10 , wherein the query of the database to identify the occurrence of the type of perception event comprises a query of scenes observed by vehicles running a given version of a perception system.

15. The computer-readable medium of claim 10 , wherein the supplemental information for the scenes comprises supplemental geometric information that indicates a spatial relationship between at least one object and one of the one or more vehicles, and wherein the query of the database is based at least in part on the supplemental geometric information.

16. The computer-readable medium of claim 10 , wherein the augmented perception data in the database is derived by:
obtaining previously-derived perception data for the scenes observed by the one or more vehicles, wherein the previously-derived perception data comprises the object-level information for the one or more objects;
deriving the supplemental information for the scenes; and
augmenting the previously-derived perception data to include the supplemental information for the scenes.

17. The computer-readable medium of claim 10 , wherein the augmented perception data comprises supplemental scene-level information for at least one scene observed by the one or more vehicles that includes one or both of spatial information or temporal information, and wherein the query is based at least in part on the supplemental scene-level information.

18. The computer-readable medium of claim 10 , wherein the one or more objects were detected within respective sequences of frames observed by respective vehicles within the scenes, and wherein maintaining the database of augmented perception data comprises storing the augmented perception data in an arrangement that encodes a hierarchical relationship between the one or more objects, the respective sequences of frames, and the scenes.

19. A computing system comprising:
at least one processor;
a non-transitory computer-readable medium; and
program instructions stored on the non-transitory computer-readabl

Assignees

• Lyft Inc

Inventors

• Jain Ashesh
• Jiang Yunjian
• Rouf Mushfiqur
• Wang Henru
• Zhang Lei

Classifications

• G08G1/165 Primary

  for passive traffic, e.g. including static obstacles, trees · CPC title

• G05D1/0219

  ensuring the processing of the whole working surface · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/95

  structured as a network, e.g. client-server architectures · CPC title

• G06V20/20

  in augmented reality scenes · CPC title