Immersive feedback loop for improving AI

Patent metadata

Field	Value
Publication number	US-11250321-B2
Application number	US-201815974651-A
Country	US
Kind code	B2
Filing date	May 8, 2018
Priority date	May 8, 2018
Publication date	Feb 15, 2022
Grant date	Feb 15, 2022

Abstract

An immersive feedback loop is disclosed for improving artificial intelligence (AI) applications used for virtual reality (VR) environments. Users may iteratively generate synthetic scene training data, train a neural network on the synthetic scene training data, generate synthetic scene evaluation data for an immersive VR experience, indicate additional training data needed to correct neural network errors indicated in the VR experience, and then generate and retrain on the additional training data, until the neural network reaches an acceptable performance level.

First claim

What is claimed is:

1. A system for training a neural network, the system comprising:
a synthetic scene generator for generating first synthetic scene data comprising a first set of synthetic sensor data and first ground truth data, and second synthetic scene data comprising a second set of synthetic sensor data and second ground truth data;
a neural network trainer for training a neural network for machine vision using the first set of synthetic sensor data and the first ground truth data;
a model evaluator for:
identifying objects in the second set of synthetic sensor data by using the neural network, and
comparing the identified objects in the second set of synthetic sensor data with the second ground truth data to determine errors in the identification of objects by the neural network; and

an immersive explorer service for:
generating a virtual reality (VR) environment using the second synthetic scene data,
indicating the errors within the VR environment;
receiving feedback indicating additional training data corresponding to the indicated errors; and
indicating the additional training data to the synthetic scene generator.

2. The system of claim 1 wherein identifying objects in the second set of synthetic sensor data by using the neural network comprises:
creating a first binary image indicating pixels showing a portion of an object, and
wherein the second ground truth data comprises a second binary image indicating pixels showing a portion of an object.

3. The system of claim 1 wherein identifying objects in the second set of synthetic sensor data by using the neural network comprises:
creating a first segmented image identifying pixels showing a portion of an object selected from a set of objects, and
wherein the second ground truth data comprises a second segmented image identifying pixels showing a portion of an object selected from the set of objects.

4. The system of claim 1 wherein the immersive explorer service sends VR viewing information to the synthetic scene generator for generating synthetic scene data for generating the VR environment.

5. The system of claim 1 wherein indicating the errors within the VR environment comprises coloring pixels corresponding to the errors with a first color.

6. The system of claim 5 wherein the immersive explorer service further indicates, within the VR environment, correct identification of objects by the neural network by coloring pixels corresponding to the correct identification with a second color different than the first color.

7. The system of claim 1 further comprising a training controller for selectively iterating a feedback loop, the feedback loop comprising:
the synthetic scene generator;
the neural network trainer;
the model evaluator; and
the immersive explorer service.

8. A method of training a neural network, the method comprising:
generating a first synthetic scene data comprising a first set of synthetic sensor data and a first ground truth data;
training a neural network for machine vision using the first synthetic scene data;
generating a second synthetic scene data comprising a second set of synthetic sensor data and a second ground truth data;
identifying objects in the second set of synthetic sensor data by using the neural network;
generating a virtual reality (VR) environment using the second synthetic scene data;
comparing the identified objects in the second set of synthetic sensor data with the second ground truth data to determine errors in the identification of objects by the neural network;
responsive to the comparison, indicating the errors within the VR environment;
receiving feedback indicating additional training data corresponding to the indicated errors;
responsive to receiving the indication of additional training data, generating a third synthetic scene data comprising a third set of synthetic sensor data and a third ground truth data, the third synthetic scene data comprising the indicated additional training data; and
training the neural network for machine vision using the third synthetic scene data.

9. The method of claim 8 wherein identifying objects in the second set of synthetic sensor data by using the neural network comprises:
creating a first binary image indicating pixels showing a portion of an object, and
wherein the second ground truth data comprises a second binary image indicating pixels showing a portion of an object.

10. The method of claim 8 wherein identifying objects in the second set of synthetic sensor data by using the neural network comprises:
creating a first segmented image identifying pixels showing a portion of an object selected from a set of objects, and
wherein the second ground truth data comprises a second segmented image identifying pixels showing a portion of an object selected from the set of objects.

11. The method of claim 8 wherein generating the second synthetic scene data comprises:
determining a VR viewing pose; and
generating the second synthetic scene data according to the VR viewing pose.

12. The method of claim 8 wherein indicating the errors within the VR environment comprises coloring pixels corresponding to the errors with a first color.

13. The method of claim 12 further comprising:
indicating, within the VR environment, correct identification of objects by the neural network by coloring pixels corresponding to the correct identification with a second color different than the first color.

14. The method of claim 12 further comprising:
iterating, until no additional synthetic scene training data is indicated through performing:
generating synthetic scene training data;
training the neural network on the synthetic scene training data;
generating synthetic scene evaluation data;
generating a VR environment using the synthetic scene evaluation data;
indicating, within the VR environment, errors in identification of objects by the neural network; and
receiving feedback indicating additional synthetic scene training data or indicating no additional synthetic scene training data.

15. One or more computer storage devices having computer-executable instructions stored thereon for training a machine vision neural network, which, on execution by a computer, cause the computer to perform operations, the instructions comprising:
a synthetic scene generator component for generating first synthetic scene data comprising a first set of synthetic sensor data and first ground truth data, and second synthetic scene data comprising a second set of synthethic sensor data and second ground truth data;
a neural network trainer component for training a neural network for machine vision using the first synthetic scene data;
a model evaluator component for:
identifying objects in the second set of synthetic sensor data by using the neural network, and
comparing the identified objects in the second set of synthetic sensor data with the second ground truth data to determine errors in the identification of objects by the neural network; and

an immersive explorer service component for:
generating a virtual reality (VR) environment using the second synthetic scene data,
indicating the errors within the VR environment;
receiving feedback indicating additional training data corresponding to the indicated errors; and
storing the additional training data in association with the second synthetic scene data.

16. The one or more computer storage devices of claim 15 wherein identifying objects in the second set of synthetic sensor data by using the neural network comprises:
creating a first binary

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Ebstyne Michael
• Escos Pedro Urbina
• Pekelny Yuri
• Shalev Emanuel

Classifications

• G06V10/945

  User interactive design; Environments; Toolboxes · CPC title

• G06V20/00

  Scenes; Scene-specific elements (control of digital cameras H04N23/60) · CPC title

• G06V10/774 Primary

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06N3/0475 Primary

  Generative networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title