Performance testing for robotic systems

Patent metadata

Field	Value
Publication number	US-2024001942-A1
Application number	US-202218273075-A
Country	US
Kind code	A1
Filing date	Jan 28, 2022
Priority date	Jan 29, 2021
Publication date	Jan 4, 2024
Grant date	—

Abstract

A computer-implemented method of modelling a perception system for perceiving objects captured in sensor data comprises: receiving a plurality of training examples, each comprising a ground truth scene for a set of sensor data and a corresponding perceived scene obtained by applying the perception system to the set of sensor data; fitting to the training examples noise model parameters, encoding a noise distribution over perceived scenes given a misdetection scene, and misdetection model parameters, encoding a misdetection distribution over misdetection scenes given a ground truth scene; computing a perception distribution over perceived scenes for a given ground truth scene by marginalizing the product of noise and misdetection distributions over multiple misdetection scenes, wherein individual objects in the ground truth scene are not associated with individual objects in the perceived scenes; fitting the noise and misdetection model parameters to match the perception distribution to the perceived scene for each training example.

First claim

1 . A computer-implemented method of modelling a perception system, the perception system for perceiving objects captured in sensor data, the method comprising:
receiving a plurality of training examples, each comprising a ground truth scene for a set of sensor data and a corresponding perceived scene obtained by applying the perception system to the set of sensor data; fitting, to the training examples, one or more noise model parameters and one or more misdetection model parameters, the noise model parameters encoding a noise distribution over possible perceived scenes given a misdetection scene, and the misdetection model parameters encoding a misdetection distribution over possible misdetection scenes given a ground truth scene; wherein a perception model distribution over possible perceived scenes is computed for the ground truth scene of each training example, by marginalizing the product of the noise distribution with the misdetection distribution over multiple possible misdetection scenes, wherein a number of objects in each of the multiple misdetection scenes is constrained to match a number of objects in the perceived scene, but the number of objects in the ground truth scene is not constrained to match a number of objects in the perceived scene, and the training example does not associate individual objects in the ground truth scene with individual objects in the corresponding perceived scene; wherein the noise and misdetection model parameters are fitted so as to substantially match the perception model distribution to the perceived scene for each training example.

2 . The method of claim 1 , wherein the noise and misdetection parameters are fitted by applying maximum likelihood estimation to match the perception model distribution to the perceived scene for each training example.

3 . The method of claim 1 , wherein the misdetection model parameters comprise one or more false positive parameters and one or more false negative parameters, the false positive parameters encoding a false positive distribution over false positive misdetections, and the false negative parameters encoding a false negative distribution over false negative misdetections, and wherein the misdetection distribution is a product of the false positive distribution and the false negative distribution.

4 . The method of claim 3 , wherein the marginalization is performed by summing the product over multiple permutations of false positive and false negative outcomes.

5 . The method of claim 1 , wherein the product is summed over a determined subset of possible misdetection scenes, the subset determined based on one or more heuristics.

6 . The method of claim 5 , wherein the subset is a determined subset of permutations of false positive and false negative outcomes.

7 . The method of claim 1 , wherein the noise model parameters encode a set of single-object noise distributions over possible object positions, each single-object noise distribution corresponding to an object of the misdetection scene, wherein the noise distribution over possible perceived scenes for a given misdetection scene is a product of the single-object noise distributions for that misdetection scene.

8 . The method of claim 1 , wherein the noise model parameters comprise one or more Gaussian noise model parameters, and the misdetection model parameters comprise one or more Gaussian noise model parameters.

9 . The method of claim 1 , wherein the noise model parameters are weights of a first neural network, and the misdetection model parameters are weights of a second neural network.

10 . The method of claim 9 , wherein the first neural network(s) predicts, in dependence on the noise model parameters, one or more noise distribution parameters of the noise distribution, and the second neural network(s) predicts, in dependence on the misdetection model parameters, one or more misdetection distribution parameters of the misdetection distribution.

11 . (canceled)

12 . The method of claim 9 , wherein the first neural network is used to generate samples of the noise distribution, and the second neural network is used to generate samples of the misdetection distribution.

13 . A computer system comprising:
memory embodying computer-readable instructions; and one or more processors coupled to the memory, the computer-readable instructions configured, when executed on the one or more hardware processors, to cause the computer system to implement the steps of:
receiving a ground truth scene comprising one or more objects;
processing the ground truth scene in a perception model to determine a perception distribution over possible perceived scenes, the perception model comprising noise parameters and misdetection parameters, the noise parameters and misdetection parameters trained to model the perception system in accordance with claim;
sampling from the perception distribution one or more times to generate one or more realistic perceived scenes for the given ground truth scene.

14 . The computer system of claim 13 , wherein the computer-readable instructions are configured to cause the computer system to sample from the perception distribution multiple times, in order to obtain multiple realistic perceived scenes with different misdetection outcomes for the same ground truth scene.

15 . The computer system of claim 14 , wherein the different misdetection outputs comprise different combinations of false positive and/or false negative object detections.

16 . The computer system of claim 13 , wherein the computer-readable instructions are configured to cause the computer system to generate the perception ground truth scene directly from a simulated scenario, such that the perception ground truth scene corresponds to an output of the perception system but is generated without applying the perception system and without the use of synthetic sensor data.

17 . The computer system of claim 16 , when applied to performance test a robotic planner in the presence of realistic perception error, wherein the robotic planner plans a trajectory for a mobile robot in the simulated scenario based on a realistic perceived scene sampled from the perception distribution.

18 . The computer system of claim 17 , when applied to performance test a planning and prediction system comprising the robotic planner and a prediction stack, wherein the robotic planner plans the trajectory based on one or more agent trajectories predicted by the prediction stack based on the realistic perceived scene.

19 . The computer system of claim 17 , comprising a test oracle configured to apply a set of predetermined rules to assess the behaviour of the mobile robot in the simulated scenario.

20 . Non-transitory computer-readable storage media having embodied thereon computer-readable instructions configured to cause, when executed on one or more hardware processors, the one or more hardware processors to implement operations comprising:
receiving a plurality of training examples, each comprising a ground truth scene for a set of sensor data and a corresponding perceived scene obtained by applying the perception system to the set of sensor data; fitting, to the training examples, one or more noise model parameters and one or more misdetection model parameters, the noise model parameters encoding a noise distribution over possible perceived scenes given a misdetection scene, and the misdetection model parameters encoding a misdetection distribution over possible misdetection scenes given a ground truth scene; w

Assignees

• Five Ai Ltd

Inventors

• Mueller Romain

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06F11/3698

  Environments for analysis, debugging or testing of software · CPC title

• B60W50/0205 Primary

  Diagnosing or detecting failures; Failure detection models · CPC title