Quantifying the Predictive Uncertainty of Neural Networks Via Residual Estimation With I/O Kernel

Patent metadata

Field	Value
Publication number	US-2020372327-A1
Application number	US-202016879934-A
Country	US
Kind code	A1
Filing date	May 21, 2020
Priority date	May 23, 2019
Publication date	Nov 26, 2020
Grant date	—

Abstract

A residual estimation with an I/O kernel (“RIO”) framework provides estimates of predictive uncertainty of neural networks, and reduces their point-prediction errors. The process captures neural network (“NN”) behavior by estimating their residuals with an I/O kernel using a modified Gaussian process (“GP”). RIO is applicable to real-world problems, and, by using a sparse GP approximation, scales well to large datasets. RIO can be applied directly to any pretrained NNs without modifications to model architecture or training pipeline.

First claim

1 . A process for training a Gaussian process (GP) to estimate residuals of a neural network (NN) model comprising:
training a NN model to make one or more predictions using a training data ( , y) input set, wherein the training data input set includes (x i ,y i )} i=1 n wherein y i are the expected outcomes by the NN model given input x i ; storing an output data set from the NN model, including the one or more predictions resulting from operation on the training data input set, wherein the output data set includes (x i ,ŷ i )} i=1 n , wherein ŷ i are the predicted outcomes by the NN model given input x i ; and training a Gaussian process (GP) to estimate residuals of the NN model when applied to raw input data x * using the training data input set (x i ,y i )} i=1 n and the output data set (x i ,ŷ i )} i=1 n .

2 . The process according to claim 1 , wherein the training of the Gaussian process (GP) includes:
calculating residuals r={r i =y i −ŷ i } i=1 n , wherein r denotes the vector of all residuals and ŷ denotes the vector of all NN model predictions; calculating an n×n covariance matrix at all pairs of training points based on a composite kernel K c (( ,ŷ),( ,ŷ)), where each entry is given by k c ((x i , ŷ i ), (x j , ŷ j ))=k in (x i , x j )+k out (ŷ i , ŷ j ), for i, j=1,2, . . . , n; and optimizing GP hyperparameters σ in 2 , l in , σ out 2 , l out and σ n 2 by maximizing log marginal likelihood

log


p


(

r
|

,

y
^

)

=

-

1
2



r
⊤



(

K
c



(

(

,

y
^

)

,

(

,

y
^

)

)

+

σ
n
2


I

)


r

-

1
2


log




K
c



(

(

,

y
^

)

,

(

,

y
^

)

)

+

σ
n
2


I



-

n
2


log



2


π
.

3 . The process according to claim 1 , wherein the NN model is a fully connected feed-forward network.

4 . A process for correcting one or more predictions of a neural network (NN) model comprising:
applying a modified Gaussian process (GP) to predictions ŷ * of a neural network (NN) model applied to raw input data x * , wherein the modified (GP) is trained using both input and output data sets from training of the NN model, the applying including: calculating residual mean; calculating residual variance; and returning distribution of calibrated prediction ŷ′ * .

5 . The process according to claim 4 , further comprises
calculating residual mean in accordance with {circumflex over ( r )} * =k * T (K c (( ,ŷ), ( ,ŷ))+σ n 2 ) −1 r; calculating residual variance in accordance with var({circumflex over (r)} * )=k c ((x * , ŷ * ))−k * T (K c (( ,ŷ), ( ,ŷ))+σ n 2 ) −1 k * ; and returning a distribution of calibrated prediction ŷ′ * in accordance with ŷ′ * ˜ (ŷ * +{circumflex over ( r )} * , var({circumflex over (r)} * )).

6 . The process according to claim 4 , wherein the NN model is a fully connected feed-forward network.

7 . The process according to claim 4 , further comprising:
training of the NN model to make one or more predictions uses a training data ( , y) input set, wherein the training data input set includes (x i , y i )} i=1 n wherein y i are the expected outcomes by the NN model given input x i ; storing an output data set from the NN m

Assignees

• Cognizant Tech Solutions U S Corporation

Inventors

• Qiu Xin
• Miikkulainen Risto
• Meyerson Elliot

Classifications

• G06N3/047 Primary

  Probabilistic or stochastic networks · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title

• G06N20/20

  Ensemble learning · CPC title

• G06N3/09

  Supervised learning · CPC title