Virtual/augmented reality data evaluation

Patent metadata

Field	Value
Publication number	US-11853390-B1
Application number	US-201816054709-A
Country	US
Kind code	B1
Filing date	Aug 3, 2018
Priority date	Aug 3, 2018
Publication date	Dec 26, 2023
Grant date	Dec 26, 2023

Abstract

Techniques for evaluating an output of a machine learning model and using the evaluation to retrain the machine learning model are described. For example, a data set that is output from a layer of the machine learning model is reduced to a 2-D or 3-D representation that is suitable for viewing. A user views the reduced data set in a viewing environment such as virtual reality or augmented reality. The user makes changes using that viewing environment. The changes are then used to retrain the machine learning model.

First claim

What is claimed is:

1. A computer-implemented method comprising:
training a neural network using a multispectral data set of vectors, each vector of the multispectral data set having a plurality of values;
generating two manifolds of reduced multidimensional vectors for intermediate representations of the multispectral data set, the intermediate representations coming from a hidden layer of the neural network;
mapping each reduced multidimensional vector of the two manifolds into a visualizable multidimensional space to generate mapped multidimensional vectors;
providing the mapped multidimensional vectors to be available for visualizing in the visualizable multidimensional space on a virtual or augmented reality device;
receiving, through a user interface, user changes to one or more of the mapped multidimensional vectors, the user changes corresponding to a changed classification of one or more mapped objects in the visualizable multidimensional space made using the visualizable multidimensional space of the virtual or augmented reality device; and
retraining the neural network using the user changes.

2. The computer-implemented method of claim 1 , wherein the two manifolds are generated using t-distributed stochastic neighbor embedding (t-SNE).

3. The computer-implemented method of claim 1 , wherein the plurality of values of the vectors of the multispectral data set is greater than three and the visualizable multidimensional space is a three-dimensional space.

4. A computer-implemented method comprising:
generating at least two manifolds of reduced multidimensional vectors for intermediate representations of a multidimensional data set output from a neural network, the intermediate representations coming from a hidden layer of the neural network;
mapping each reduced multidimensional vector of the at least two manifolds into a visualizable multidimensional space to generate mapped multidimensional vectors;
providing the mapped multidimensional vectors to be available for visualizing in the visualizable multidimensional space;
receiving, through a user interface, changes to one or more of the mapped multidimensional vectors, the changes corresponding to a changed classification of one or more mapped objects in the visualizable multidimensional space made during visualization of the one or more mapped objects in the visualizable multidimensional space; and
retraining the neural network using the changes.

5. The computer-implemented method of claim 4 , wherein the visualizable multidimensional space is a three-dimensional space.

6. The computer-implemented method of claim 4 , wherein the manifolds are generated using t-distributed stochastic neighbor embedding (t-SNE).

7. The computer-implemented method of claim 4 , wherein the manifolds are generated using principal component analysis.

8. The computer-implemented method of claim 4 , further comprising post-processing the intermediate representations of the multidimensional data set to remove at least one dimension prior to generating the manifolds.

9. The computer-implemented method of claim 4 , further comprising post-processing the intermediate representations of the multidimensional data set to normalize at least one dimension prior to generating the manifolds.

10. The computer-implemented method of claim 4 , wherein the neural network is a convolutional neural network.

11. The computer-implemented method of claim 4 , wherein visualizing in the visualizable multidimensional space is provided using augmented reality.

12. The computer-implemented method of claim 4 , wherein visualizing in the visualizable multidimensional space is provided using virtual reality.

13. The computer-implemented method of claim 4 , receiving a request to perform virtual or augmented reality data manipulation, the request including a location of the multidimensional data set, a location of a model to train, and a location to store the at least two manifolds.

14. A system comprising:
a visualization device including memory and one or more processors, and configured for viewing a multidimensional space; and
a data evaluation system coupled to the visualization device, the data evaluation system including memory storing instructions that, when executed by one or more processors of the data evaluation system cause the data evaluation system to:
generate at least two manifolds of reduced multidimensional vectors for intermediate representations of a multidimensional data set output from a neural network, the intermediate representations coming from a hidden layer of the neural network;
map each reduced multidimensional vector of the at least two manifolds into a visualizable multidimensional space to generate mapped multidimensional vectors;
provide the mapped multidimensional vectors to be available for visualizing in the visualizable multidimensional space using the visualization device;
receive, via a user interface of the visualization device, changes to one or more of the mapped multidimensional vectors, the changes corresponding to a changed classification of one or more mapped objects in the visualizable multidimensional space made during visualization of the one or more mapped objects in the visualizable multidimensional space by the visualization device; and
retrain the neural network using the changes to one or more of the mapped multidimensional vectors.

15. The system of claim 14 , wherein the visualization device is an augmented reality device.

16. The system of claim 14 , wherein the visualization device is a virtual reality device.

17. The system of claim 14 , wherein the visualizable multidimensional space is a three-dimensional space.

18. The system of claim 14 wherein the manifolds are generated using t-distributed stochastic neighbor embedding (t-SNE).

19. The system of claim 14 , wherein the manifolds are generated using principal component analysis.

20. The system of claim 14 , wherein the neural network is a convolutional neural network.

Assignees

• Amazon Tech Inc

Inventors

• Bowman Bradley Scott
• Lapin Maksim
• Dirac Leo Parker

Classifications

• G06F18/2135 Primary

  based on approximation criteria, e.g. principal component analysis · CPC title

• G06F16/904

  Browsing; Visualisation therefor (for navigating the web G06F16/954; browsing optimisation for the web G06F16/957) · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06N3/08 Primary

  Learning methods · CPC title