System and methodology for performance verification of multi-agent autonomous robotic systems

Patent metadata

Field	Value
Publication number	US-2021237772-A1
Application number	US-201915734865-A
Country	US
Kind code	A1
Filing date	May 12, 2019
Priority date	Jun 5, 2018
Publication date	Aug 5, 2021
Grant date	—

Abstract

A computerized method of performing safety and functional verification of algorithms, for control of autonomous vehicles, comprises: iteratively performing an adjustment, the adjustment comprising at least one of the following: (i) updating the value of parameter(s) indicative of noise and/or delay in simulated sensor(s), associated with a computerized simulation framework corresponding to simulated autonomous vehicle(s) and to operational environment(s), by increasing noise and/or delay; and (ii) updating the value of parameter(s) indicative of noise and/or delay in a response of the simulated autonomous vehicle(s) to command(s), by increasing the noise and/or delay. This is done until obtaining from the computerized simulation framework an increased- severity computerized simulation framework. The increased-severity computerized simulation framework meets a criterion that can be utilized for statistical safety verification and/or statistical functional performance verification of the algorithm(s).

First claim

1 - 37 . (canceled)

38 . A computerized method of performing safety and functional verification of algorithms for control of autonomous vehicles, the computerized method comprising:
iteratively performing an adjustment, said adjustment comprising at least one of the following:
i) updating the value of at least one parameter indicative of at least one of noise or delay in at least one simulated sensor, associated with a computerized simulation framework corresponding to at least one simulated autonomous vehicle and to at least one operational environment, by increasing the at least one of noise or delay, or
ii) updating the value of at least one parameter indicative of at least one of noise or delay in a response of the at least one simulated autonomous vehicle to at least one command, by increasing the at least one of noise or delay until obtaining from the computerized simulation framework an increased-severity computerized simulation framework,
wherein the increased-severity computerized simulation framework meets a criterion that can be utilized for at least one of statistical safety verification or statistical functional performance verification of the at least one algorithm.

39 . The computerized method of claim 38 , wherein the iterative performing of an adjustment comprises:
(a) providing at least one algorithm for control of at least one autonomous vehicle; (b) providing the computerized simulation framework, wherein the computerized simulation framework interacts with the at least one algorithm, by at least providing simulated sensor data to the algorithm and receiving commands from the algorithm, t he computerized simulation framework including the at least one parameter indicative of at least one of noise or delay in the at least one simulated sensor, the computerized simulation framework including the at least one parameter indicative of at least one of noise or delay in the response of the at least one simulated autonomous vehicle to the least one command, (c) providing at least one set of parameters of the computerized simulation framework indicative of navigation scenarios; (d) providing at least one set of calibration criteria indicative of at least one of algorithm performance requirements or algorithm safety requirements; (e) providing at least one set of statistical calibration criteria indicative of the at least one set of calibration criteria; (f) performing the updating of said step (i) and the updating of said step (ii), wherein the values of the at least one parameter indicative of at least one of noise or delay in at least one sensor and of the at least one parameter indicative of at least one of noise or delay in a response of the at least one autonomous vehicle after said updating constituting a current set of noise and delay values; (g) generating a statistically significant number of calibration scenarios, based on at least one set of parameters of the computerized simulation framework indicative of navigation scenarios; (h) running the statistically significant number of calibration scenarios in the computerized simulation framework, based on the current set of noise and delay values, thereby generating first results; (i) determining whether the first results meet the at least one set of statistical calibration criteria; (j) in response to the first results meeting the at least one set of statistical calibration criteria, setting the current values of the at least one parameter indicative of at least one of noise or delay in at least one sensor and of the at least one parameter indicative of at least one of noise or delay in a response of the at least one autonomous vehicle to constitute a previous set of noise and delay values, and recording the previous set of noise and delay values in a list of previous sets of noise and delay values; (k) repeatedly performing said steps (f) to (j) until the first results do not meet the at least one set of statistical calibration criteria; (l) setting the current set of noise and delay values to constitute a failed set of noise and delay values; and (m) selecting a set of noise and delay values, which is less noisy than the failed set of noise and delay values, and setting the selected set of noise and delay values to constitute the current set of noise and delay values; wherein the increased-severity computerized simulation framework is based on the computerized simulation framework and on the current set of noise and delay values; and wherein the criterion that can be utilized for at least one of statistical safety verification or statistical functional performance verification is whether the first results meet the at least one set of statistical calibration criteria.

40 . The computerized method of claim 39 , wherein the statistically significant number of calibration scenarios is significantly smaller than a statistically significant number of algorithm verification test scenarios.

41 . The computerized of claim 39 , wherein said selecting a set of noise and delay values comprises selecting, from the list of previous sets of noise and delay values, one of the previous sets of noise and delay values.

42 . The computerized method of claim 41 , wherein the one of the previous sets of noise and delay values comprises a then-current previous set of noise and delay values.

43 . The computerized method of claim 38 , wherein the at least one command is an actuator command.

44 . The computerized method of claim 38 , wherein the at least one sensor is a sensor associated with the vehicle.

45 . The computerized method of claim 38 , wherein the at least one sensor is a fixed sensor.

46 . The computerized method of claim 38 , wherein the statistically significant number of calibration scenarios are pseudo-random calibration scenarios.

47 . The computerized method of claim 37 , further comprising:
(n) providing at least one set of criteria indicative of algorithm performance requirements and of algorithm safety requirements, constituting at least one set of algorithm verification test criteria; (o) providing at least one set of statistical verification criteria, indicative of the at least one set of algorithm verification test criteria; (p) generating said statistically significant number of algorithm verification test scenarios, based on the at least one set of parameters of the computerized simulation framework indicative of navigation scenarios, said statistically significant number of algorithm verification test scenarios constituting an algorithm verification scenarios set; (q) running the algorithm verification scenario set on the computerized simulation framework, thereby generating a set of second results; (r) determining whether the second results meet the at least one set of statistical verification criteria; (s) in response to the second results meeting the at least one set of statistical verification criteria, generating a report indicating compliance of the at least one algorithm to the at least one set of statistical verification criteria. (t) in response to the second results not meeting the at least one set of statistical verification criteria, generating a report indicating possible non-compliance of the at least one algorithm to the at least one set of statistical verification criteria.

48 . The computerized method according to claim 38 , wherein at least one of the first results or the second results serving as inputs to machine learning in updating of the algorithm.

49 . The computerized method of claim 47 , further comprising:
(u) responsive to determining that at least one second result in the set of second results does not meet the at leas

Assignees

• Elta Systems Ltd

Inventors

• Meltz Daniel
• Sirkis Amit

Classifications

• G06F30/20 Primary

  Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30) · CPC title

• B60W60/0015 Primary

  specially adapted for safety · CPC title

• G06F18/2415

  based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate · CPC title

• G09B9/00

  Simulators for teaching or training purposes (for the use of weapons F41; computing aspects G06; {protocols for games, networked simulations or virtual reality H04L67/131}) · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title