System and methodology for performance verification of multi-agent autonomous robotic systems

Patent metadata

Field	Value
Publication number	US-12099782-B2
Application number	US-201915734865-A
Country	US
Kind code	B2
Filing date	May 12, 2019
Priority date	Jun 5, 2018
Publication date	Sep 24, 2024
Grant date	Sep 24, 2024

Abstract

A computerized method of performing safety and functional verification of algorithms, for control of autonomous vehicles, comprises: iteratively performing an adjustment, the adjustment comprising at least one of the following: (i) updating the value of parameter(s) indicative of noise and/or delay in simulated sensor(s), associated with a computerized simulation framework corresponding to simulated autonomous vehicle(s) and to operational environment(s), by increasing noise and/or delay; and (ii) updating the value of parameter(s) indicative of noise and/or delay in a response of the simulated autonomous vehicle(s) to command(s), by increasing the noise and/or delay. This is done until obtaining from the computerized simulation framework an increased-severity computerized simulation framework. The increased-severity computerized simulation framework meets a criterion that can be utilized for statistical safety verification and/or statistical functional performance verification of the algorithm(s).

First claim

The invention claimed is:

1. A computerized method of performing safety and functional verification of algorithms for control of autonomous vehicles, configured for performance by a processing circuitry that comprises a processor and a memory, the method the computerized method comprising performing the following:
A. providing at least one algorithm for control of at least one autonomous vehicle;
B. providing a computerized simulation framework, wherein the computerized simulation framework interacts with the at least one algorithm by at least providing simulated sensor data to the algorithm and receiving commands from the algorithm,
the computerized simulation framework corresponding to the at least one simulated autonomous vehicle and to at least one operational environment,
the computerized simulation framework including at least one of the following: at least one parameter indicative of at least one of noise or delay in at least one simulated sensor associated with the computerized simulation framework, and at least one parameter indicative of at least one of noise or delay in a response of the at least one simulated autonomous vehicle to least one command;

C. providing at least one set of parameters of the computerized simulation framework indicative of navigation scenarios;
D. providing at least one set of calibration criteria indicative of at least one of algorithm performance requirements or algorithm safety requirements;
E. providing at least one set of statistical calibration criteria indicative of the at least one set of calibration criteria; and
F. iteratively performing an adjustment, until obtaining from the computerized simulation framework an increased-severity computerized simulation framework,
said adjustment comprising at least one of the following:
i) updating a value of the at least one parameter indicative of the at least one of noise or delay in the at least one simulated sensor, by increasing the at least one of noise or delay; or
ii) updating a value of the at least one parameter indicative of the at least one of noise or delay in the response of the at least one simulated autonomous vehicle to the at least one command, by increasing the at least one of noise or delay,

wherein the iterative performing of the adjustment comprises:
(a) performing at least one of the updating of said step (i) and the updating of said step (ii), wherein, after said updating, the updated values of the at least one parameter indicative of the at least one of noise or delay in the at least one sensor, and the updated values of the at least one parameter indicative of the at least one of noise or delay in the response of the at least one autonomous vehicle, constitute a current set of at least one of noise values and delay values;
(b) generating a number of calibration scenarios, based on at least one set of parameters of the computerized simulation framework indicative of navigation scenarios;
(c) running the number of calibration scenarios in the computerized simulation framework, based on the current set of at least one of noise values and delay values, thereby generating first results;
(d) determining whether the first results meet the at least one set of statistical calibration criteria;
(e) in response to the first results meeting the at least one set of statistical calibration criteria, performing the following:
(I) setting the current set of at least one of noise values and delay values to constitute a previous set of at least one of noise values and delay values;

(f) repeatedly performing said steps (a) to (e) until the first results do not meet the at least one set of statistical calibration criteria;
(g) setting the current set of at least one of noise values and delay values to constitute a failed set of at least one of noise values and delay values;
(h) selecting a set of at least one of noise values and delay values, which is less noisy than the failed set of at least one of noise values and delay values; and
(i) setting the selected set of at least one of noise values and delay values to constitute the current set of at least one of noise values and delay values,
wherein the increased-severity computerized simulation framework is based on the computerized simulation framework and on the current set of at least one of noise values and delay values,
wherein the increased-severity computerized simulation framework meets a criterion that can be utilized for at least one of statistical safety verification and statistical functional performance verification of the at least one algorithm, wherein the criterion that can be utilized for at least one of statistical safety verification or statistical functional performance verification is whether the first results meet the at least one set of statistical calibration criteria.

2. The computerized method of claim 1 , wherein said step (e) further comprises:
(II) recording the previous set of at least one of noise values and delay values in a list of previous sets of at least one of noise values and delay values.

3. The computerized method of claim 1 , wherein said selecting a set of noise and delay values comprises selecting, from the list of previous sets of noise and delay values, one of the previous sets of noise and delay values.

4. The computerized method of claim 3 , wherein the one of the previous sets of noise and delay values comprises a then-current previous set of at least one of noise values and delay values.

5. The computerized method of claim 1 , wherein the at least one command is an actuator command.

6. The computerized method of claim 1 , wherein the at least one sensor is a sensor associated with the vehicle.

7. The computerized method of claim 1 , wherein the at least one sensor is a fixed sensor.

8. The computerized method of claim 1 , wherein the number of calibration scenarios are pseudo-random calibration scenarios.

9. The computerized method of claim 5 , further comprising:
(j) providing at least one set of criteria indicative of algorithm performance requirements and of algorithm safety requirements, constituting at least one set of algorithm verification test criteria;
(k) providing at least one set of statistical verification criteria, indicative of the at least one set of algorithm verification test criteria;
(l) generating said number of algorithm verification test scenarios, based on the at least one set of parameters of the computerized simulation framework indicative of navigation scenarios, said number of algorithm verification test scenarios constituting an algorithm verification scenarios set;
(m) running the algorithm verification scenario set on the computerized simulation framework, thereby generating a set of second results;
(n) determining whether the second results meet the at least one set of statistical verification criteria;
(o) in response to the second results meeting the at least one set of statistical verification criteria, generating a report indicating compliance of the at least one algorithm to the at least one set of statistical verification criteria;
(p) in response to the second results not meeting the at least one set of statistical verification criteria, generating a report indicating possible non-compliance of the at least one algorithm to the at least one set of statistical verification criteria.

10. The computerized method of claim 9 , further comprising:
(q) responsive to determining that at least one second result in the set of second results does not meet the at least one set of algorithm verification test criteria, selecting the at least one second result in the set of second results that does not meet the at least one set of algorithm verification test criteria;

Assignees

• Elta Systems Ltd

Inventors

• Meltz Daniel
• Sirkis Amit

Classifications

• B60W2420/408

  Radar; Laser, e.g. lidar · CPC title

• G06F18/2415

  based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• B60W2420/403

  Image sensing, e.g. optical camera · CPC title

• B60W30/08 Primary

  Active safety systems} predicting or avoiding probable or impending collision {or attempting to minimise its consequences · CPC title