Perception error models

Patent metadata

Field	Value
Publication number	US-11734473-B2
Application number	US-201916708019-A
Country	US
Kind code	B2
Filing date	Dec 9, 2019
Priority date	Sep 27, 2019
Publication date	Aug 22, 2023
Grant date	Aug 22, 2023

Abstract

Techniques for determining an error model based on vehicle data and ground truth data are discussed herein. To determine whether a complex system (which may be not capable of being inspected) is able to operate safely, various operating regimes (scenarios) can be identified based on operating data. To provide safe operation of such a system, an error model can be determined that can provide a probability associated with perception data and a vehicle can determine a trajectory based on the probability of an error associated with the perception data.

First claim

What is claimed is:

1. A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause the system to perform operations comprising:
receiving vehicle data from a vehicle, the vehicle data associated with a state of an object;
receiving ground truth data associated with the object;
determining, based at least in part on the vehicle data and the ground truth data, an error;
determining, based at least in part on the vehicle data, a plurality of parameters;
clustering, as a plurality of clusters and based at least in part on the plurality of parameters and the error, at least a portion of the vehicle data, the clustering comprising associating a distribution of the error with a cluster of the plurality of clusters;
determining classification data identifying a classification of the object;
determining object data identifying an object parameter of the object;
determining an error distribution associated with at least one of a first cluster of the plurality of clusters or a second cluster of the plurality of clusters, the first cluster associated with the classification of the object and the second cluster associated with the object parameter of the object; and
determining, based at least in part on the portion of the vehicle data associated with a cluster of the plurality of clusters, an error model, wherein the error model includes an error distribution associated with at least one of a true positive error or a false positive error.

2. The system of claim 1 , wherein the vehicle data is based at least in part on sensor data from a sensor associated with the vehicle.

3. The system of claim 1 , wherein the plurality of parameters is associated with at least two or more of a weather condition, a first time of day, a second time of year, a distance to the object, the classification of the object, a size of the object, a velocity of the object, a position of the object, or an orientation of the object.

4. The system of claim 1 , wherein the error is a first error, the operations further comprising:
receiving perception data;
determining, based at least in part on the perception data and the error model, a second error associated with the perception data; and
controlling, based at least in part on the perception data and the second error, the vehicle.

5. A method comprising:
receiving vehicle data from a vehicle, the vehicle data associated with a state of an object;
receiving, based at least in part on the vehicle data, ground truth data associated with the object;
determining, based at least in part on the vehicle data and the ground truth data, an error;
determining, based at least in part on the vehicle data, a parameter;
clustering, as a plurality of clusters and based at least in part on the parameter and the error, a portion of the vehicle data, the clustering comprising associating a distribution of the error with a cluster of the plurality of clusters;
determining classification data identifying a classification of the object;
determining object data identifying an object parameter of the object
determining an error distribution associated with at least one of a first cluster of the plurality of clusters or a second cluster of the plurality of clusters, the first cluster associated with the classification of the object and the second cluster associated with the object parameter of the object; and
determining, based at least in part on the portion of the vehicle data associated with a cluster of the plurality of clusters, an error model, wherein the error model includes an error distribution associated with at least one of a true positive error or a false positive error.

6. The method of claim 5 , wherein the state of the object comprises at least one of a size of the object, a location of the object, an orientation of the object, a velocity of the object, or a position of the object.

7. The method of claim 5 , further comprising:
determining, based at least in part on the vehicle data and the ground truth data, an occurrence frequency associated with the error; and
determining, based at least in part on the occurrence frequency, the error distribution included in the error model.

8. The method of claim 5 , further comprising:
receiving simulation data associated with a simulated vehicle controller in a simulated environment;
determining, based at least in part on the error model and the simulation data, perturbed simulation data;
sending the perturbed simulation data to the simulated vehicle controller in the simulated environment; and
determining, based at least in part on the perturbed simulation data, a response indicating how the simulated vehicle controller responds to the perturbed simulation data.

9. The method of claim 8 , wherein:
the error is a first error;
the simulation data comprises a classification associated with a simulated object represented in the simulated environment; and
the perturbed simulation data indicates a second error associated with at least one of a position of the simulated object, an orientation of the simulated object, an extent of the simulated object, or a velocity of the simulated object.

10. The method of claim 5 , further comprising:
determining, at a first time, a first error distribution associated with a false negative error;
determining, at a second time after the first time, a second error distribution associated with the true positive error; and
determining, at a third time after the second time, a third error distribution associated with the false positive error,
wherein the error model further comprises the first error distribution, the second error distribution, and the third error distribution.

11. The method of claim 5 , further comprising:
determining a cost associated with fitting the vehicle data to the error model; and
determining, based at least in part on the cost, the error model.

12. One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:
receiving vehicle data from a vehicle, the vehicle data associated with a state of an object;
receiving ground truth data associated with the object;
determining, based at least in part on the vehicle data and the ground truth data, an error;
determining, based at least in part on the vehicle data, a parameter;
clustering, as a plurality of clusters and based at least in part on the parameter, a portion of the vehicle data, the clustering comprising associating a distribution of the error with a cluster of the plurality of clusters;
determining classification data identifying a classification of the object;
determining object data identifying an object parameter of the object
determining an error distribution associated with at least one of a first cluster of the plurality of clusters or a second cluster of the plurality of clusters, the first cluster associated with the classification of the object and the second cluster associated with the object parameter of the object; and
determining, based at least in part on the portion of the vehicle data associated with a cluster of the plurality of clusters, an error model, wherein the error model includes an error distribution associated with at least one of a true positive error or a false positive error.

13. The one or more non-transitory computer-readable media of claim 12 , the operations further comprising:
determining, based at least in part on the vehicle data and the ground truth d

Assignees

• Zoox Inc

Inventors

• Modalavalasa Sai Anurag
• Bagschik Gerrit
• Crego Andrew Scott
• Deux Antoine Ghislain
• Lyasoff Rodin
• Philbin James William Vaisey
• Rege Ashutosh Gajanan
• Reschka Andreas Christian
• Wimmershoff Marc

Classifications

• G05D1/0214

  in accordance with safety or protection criteria, e.g. avoiding hazardous areas (monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13) · CPC title

• G05D1/0088

  characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00) · CPC title

• G05D1/0027

  involving a plurality of vehicles, e.g. fleet or convoy travelling (fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00) · CPC title

• G06F30/20 Primary

  Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30) · CPC title

• G06F18/23

  Clustering techniques · CPC title