Approaches for encoding environmental information

Patent metadata

Field	Value
Publication number	US-12049229-B2
Application number	US-202016817379-A
Country	US
Kind code	B2
Filing date	Mar 12, 2020
Priority date	Jun 28, 2019
Publication date	Jul 30, 2024
Grant date	Jul 30, 2024

Abstract

Systems, methods, and non-transitory computer-readable media can determine sensor data captured by at least one sensor of a vehicle while navigating an environment over a period of time. Information describing one or more agents associated with the environment during the period of time can be determined based at least in part on the captured sensor data. A schema-based encoding describing the environment during the period of time can be generated based at least in part on the determined information and a scenario schema, wherein the schema-based encoding provides a structured representation of the environment during the period of time.

First claim

What is claimed is:

1. A computer-implemented method comprising: determining, by a computing system, a particular scenario of an environment based on sensor data associated with the environment captured by at least one vehicle, wherein the sensor data describes motions of at least one agent and actions performed by the at least one agent during a period of time, and wherein the particular scenario represents the at least one agent included in the environment and a motion associated with the at least one agent;
(i) determining, by the computing system, the particular scenario based on a first motion associated with the at least one agent during a first period of time within the period of time;
comparing, by the computing system, at least one first feature describing the particular scenario during the first period of time to a first set of features associated with a high-level scenario classification of a scenario group and a second set of features associated with a lower-level scenario classification of the scenario group; and
determining, by the computing system, that at least one first feature describing the particular scenario during the first period of time corresponds to the lower-level scenario classification of the scenario group based on a comparison of the at least one first feature and the second set of features associated with the lower-level scenario classification of the scenario group; or
(ii) determining, by the computing system, the particular scenario based on a second motion associated with the at least one agent during a second period of time within the period of time;
comparing, by the computing system, at least one second feature describing the particular scenario during the second period of time to a first set of features associated with a high-level scenario classification of a scenario group and a second set of features associated with a lower-level scenario classification of the scenario group; and
determining, by the computing system, that at least one second feature describing the particular scenario during the second period of time corresponds to the high-level scenario classification of the scenario group based on a comparison of the at least one second feature and the first set of features associated with the high-level scenario classification of the scenario group;
determining, by the computing system, according to a similarity threshold, that the at least one feature describing the particular scenario corresponds to either the first set of features associated with the high-level scenario classification or the second set of features associated with the lower-level scenario classification;
classifying, by the computing system, in accordance with the similarity threshold, the particular scenario according to the high-level scenario classification of the scenario group or the lower-level scenario classification of the scenario group; and
providing, by the computing system, information describing the classified particular scenario; wherein causing, by the computing system, modification of an operation of the at least one vehicle in the environment based on the information describing the classified particular scenario.

2. The computer-implemented method of claim 1 , wherein the high-level classification corresponds to at least one of a scenario family or a scenario sub-family, and the lower-level classification corresponds to at least one of a scenario sub-family or an individual scenario.

3. The computer-implemented method of claim 1 , wherein the high-level scenario classification of the scenario group and the lower-level scenario classification of the scenario group are representative of a multi-level taxonomy of the scenario group.

4. The computer-implemented method of claim 1 , wherein determining the particular scenario of the environment comprises:
generating, by the computing system, a schema-based encoding based on the sensor data associated with the environment; and
determining, by the computing system, the particular scenario of the environment based on the schema-based encoding.

5. The computer-implemented method of claim 4 , wherein the schema-based encoding represents the at least one agent and the motion associated with the at least one agent over a period of time.

6. The computer-implemented method of claim 4 , wherein the schema-based encoding represents at least one of: map features associated with the environment, a zone associated with the environment, contextual information associated with the environment, or weather conditions associated with the environment.

7. The computer-implemented method of claim 1 , wherein, classifying the particular scenario according to the high-level scenario classification of the scenario group or the lower-level scenario classification of the scenario group further comprises:
classifying, by the computing system, the particular scenario according to the high-level scenario classification based on a determination that the at least one feature describing the particular scenario corresponds to the first set of features associated with the high-level scenario classification in accordance with the similarity threshold; or
classifying, by the computing system, the particular scenario according to the lower-level scenario classification based on a determination that the at least one feature describing the particular scenario corresponds to the second set of features associated with the lower-level scenario classification in accordance with the similarity threshold.

8. The computer-implemented method of claim 1 wherein the motions of the at least one agent are described by one or more of: trajectories, distances, velocities, directions, geographic locations, or semantic locations.

9. The computer-implemented method of claim 1 , wherein, subsequent to classifying the particular scenario as the high-level scenario classification, the classified particular scenario is associated with the first set of features, or subsequent to classifying the particular scenario as the lower-level scenario classification, the classified particular scenario is associated with the second set of features.

10. A system comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform:
determining a particular scenario of an environment based on sensor data associated with the environment captured by at least one vehicle, wherein the sensor data describes motions of at least one agent and actions performed by the at least one agent during a period of time, and wherein the particular scenario represents the at least one agent included in the environment and a motion associated with the at least one agent;
(i) determining the particular scenario based on a first motion associated with the at least one agent during a first period of time within the period of time;
comparing at least one first feature describing the particular scenario during the first period of time to a first set of features associated with a high-level scenario classification of a scenario group and a second set of features associated with a lower-level scenario classification of the scenario group; and
determining that at least one first feature describing the particular scenario during the first period of time corresponds to the lower-level scenario classification of the scenario group based on a comparison of the at least one first feature and the second set of features associated with the lower-level scenario classification of the scenario group; or
(ii) determining the particular scenario based on a second motion associated with the at least one agent during a second period of time within the period of time;
compar

Assignees

• Lyft Inc

Inventors

• Dong Lina
• Hou Weiyi
• Khandelwal Somesh
• Kirigin Ivan
• Li Shaojing
• Liu Ying
• Lu David Tse-Zhou
• Pinkerton Robert Charles Kyle
• Shet Vinay
• Sun Shaohui

Classifications

• B60W2555/20

  Ambient conditions, e.g. wind or rain · CPC title

• G06F16/212

  with details for data modelling support · CPC title

• B60W30/0956

  the prediction being responsive to traffic or environmental parameters · CPC title

• B60W60/00272

  relying on extrapolation of current movement · CPC title

• B60W2554/4044

  Direction of movement, e.g. backwards · CPC title