Neural network memory computing system and method

Patent metadata

Field	Value
Publication number	US-10929612-B2
Application number	US-201816217804-A
Country	US
Kind code	B2
Filing date	Dec 12, 2018
Priority date	Apr 24, 2018
Publication date	Feb 23, 2021
Grant date	Feb 23, 2021

Abstract

Provided are a neural network memory computing system and method. The neural network memory computing system includes a first processor configured to learn a sense-making process on the basis of sense-making multimodal training data stored in a database, receive multiple modalities, and output a sense-making result on the basis of results of the learning, and a second processor configured to generate a sense-making training set for the first processor to increase knowledge for learning a sense-making process and provide the generated sense-making training set to the first processor.

First claim

What is claimed is:

1. A neural network memory computing system for increasing knowledge for grasping a meaning of input multiple modalities, the system comprising:
a first processor configured to learn a sense-making process based on sense-making multimodal training data stored in a database, receive multiple modalities, and output a sense- making result based on results of the learning; and
a second processor configured to generate a sense-making training set for the first processor to increase knowledge for learning a sense-making process and provide the generated sense-making training set to the first processor,
wherein the first processor embeds the input multiple modalities including a query and associated information in units of vectors having a preset magnitude, repeatedly inputs the embedded multiple modalities including the query and associated information in a deep neural network (DNN), and
wherein using a weight output from the DNN, the first processor updates a piece of data whose sense-making result is related to associated information corresponding to a query and answer for sense making among pieces of data in a memory space in which the associated information is stored.

2. The neural network memory computing system of claim 1 , wherein the first processor embeds the sense-making multimodal training data in units of vectors having a preset magnitude and generates a sense-making training set based on the vectors.

3. The neural network memory computing system of claim 2 , wherein the first processor learns a process of extracting features of visual and auditory pieces of the sense-making multimodal training data using a deep neural network (DNN), outputs final hidden-layer values of the DNN for the visual and auditory pieces of sense-making training data, and embeds the final hidden-layer values in the vectors.

4. The neural network memory computing system of claim 2 , wherein, for a piece of the sense-making multimodal training data corresponding to text, the first processor learns a deep neural network (DNN)-based word vector space using a previously provided text corpus, outputs word vector values of the text, and embeds the word vector values in the vectors.

5. The neural network memory computing system of claim 2 , wherein when associated information corresponding to a query and answer for sense making is not stored in a memory space, the first processor performs tagging for writing the associated information in the memory space and tags the input query as a sense-making training input and a sense-making training output to generate the sense-making training set for the query and answer.

6. The neural network memory computing system of claim 5 , wherein the first processor classifies input tagged for writing as a text modality, a visual modality, and an auditory modality among multiple modalities included in the sense-making training set and stores the classified input in the memory space.

7. The neural network memory computing system of claim 6 , wherein the first processor sets modalities tagged as the sense-making training input among the multiple modalities included in the sense-making training set as an input to a deep neural network (DNN), sets modalities tagged as the sense-making training output as an output of the DNN, and learns the DNN.

8. The neural network memory computing system of claim 1 , wherein the first processor outputs the sense-making result.

9. The neural network memory computing system of claim 1 , wherein the second processor samples domain information and learns a query generation deep neural network (DNN) which generates the sense-making training set using a domain query-answer set constructed based on the sampled domain information.

10. The neural network memory computing system of claim 9 , wherein the second processor calls a text input corresponding to an answer and modality information whose associated information is related to the text input, converts the text input and the modality information into vectors, sets the vectors as input to the query generation DNN, sets text converted into a vector corresponding to a query as an output of the query generation DNN, and learns the query generation DNN.

11. A neural network memory computing method for increasing knowledge for grasping a meaning of input multiple modalities, the method comprising:
embedding sense-making multimodal training data stored in a database in vectors having a preset magnitude;
generating a sense-making training set based on the vectors;
learning a deep neural network (DNN) based on multiple modalities included in the generated sense-making training set; and
outputting results of grasping a meaning of input multiple modalities based on the learned DNN,
wherein the embedding of the sense-making multimodal training data in the vectors having the preset magnitude comprises learning a process of extracting features of visual and auditory pieces of the sense-making multimodal training data using the DNN, outputting final hidden-layer values of the DNN for the visual and auditory pieces of sense-making training data.

12. The neural network memory computing method of claim 11 , wherein the embedding of the sense-making multimodal training data in the vectors having the preset magnitude comprises embedding the final hidden-layer values in the vectors.

13. The neural network memory computing method of claim 11 , wherein the embedding of the sense-making multimodal training data in the vectors having the preset magnitude comprises, for a piece of the sense-making multimodal training data corresponding to text, learning a DNN-based word vector space using a previously provided text corpus, outputting word vector values of the text, and embedding the word vector values in the vectors.

14. The neural network memory computing method of claim 13 , wherein the generating of the sense-making training set based on the vectors comprises:
determining whether associated information corresponding to a query and answer for sense making is stored in a memory space;
when it is determined that the associated information is not stored in the memory space, performing tagging for writing the associated information in the memory space; and
tagging an input query as a sense-making training input and a sense-making training output to generate the sense-making training set for the query and answer.

15. The neural network memory computing method of claim 14 , wherein the learning of the DNN based on the multiple modalities included in the generated sense-making training set comprises:
classifying input tagged for writing as a text modality, a visual modality, and an auditory modality among the multiple modalities included in the sense-making training set and storing the classified input in the memory space;
setting modalities tagged as the sense-making training input among the multiple modalities included in the sense-making training set as input to the DNN; and
setting modalities tagged as the sense-making training output as output of the DNN and learning the DNN.

16. The neural network memory computing method of claim 11 , wherein the outputting of the results of grasping a meaning of the input multiple modalities based on the learned DNN comprises:
embedding the multiple modalities including an input query and associated information in units of vectors having the preset magnitude; and
repeatedly inputting the embedded multiple modalities including the query and associated information in the DNN and outputting sense-making results.

17. The neural network memory computing method of

Assignees

• Electronics & Telecommunications Res Inst

Inventors

• Jung Ho Young
• Kim Hyun Woo
• Song Hwa Jeon
• Chung Eui Sok
• Park Jeon Gue

Classifications

• G06N3/042

  Knowledge-based neural networks; Logical representations of neural networks · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0499

  Feedforward networks · CPC title