Neural network fusion apparatus and modular neural network fusion method and matching interface generation method for the same

Patent metadata

Field	Value
Publication number	US-2020065654-A1
Application number	US-201916535709-A
Country	US
Kind code	A1
Filing date	Aug 8, 2019
Priority date	Aug 22, 2018
Publication date	Feb 27, 2020
Grant date	—

Abstract

Disclosed herein are a neural network fusion apparatus and a modular neural network fusion method and a matching interface generation method of the neural network fusion apparatus. The modular neural network fusion method in a neural network fusion apparatus includes collecting meta-information of a first neural network and a second neural network, connecting a matching module between the first neural network and the second neural network using the collected meta-information so that output data of the first neural network matches input data of the second neural network, and executing a neural network interworking service for obtaining output data of the second neural network from input data of the first neural network through the first and second neural networks connected to each other via the matching module.

First claim

What is claimed is:

1 . A modular neural network fusion method in a neural network fusion apparatus, comprising:
collecting meta-information of a first neural network and a second neural network; connecting a matching module between the first neural network and the second neural network using the collected meta-information so that output data of the first neural network matches input data of the second neural network; and executing a neural network interworking service for obtaining output data of the second neural network from input data of the first neural network through the first and second neural networks connected to each other via the matching module.

2 . The modular neural network fusion method of claim 1 , wherein the collecting the meta-information comprises:
reading information about a data type and format information of an output parameter from a meta-information storage of the first neural network; and reading information about a data type and format information of an input parameter from a meta-information storage of the second neural network.

3 . The modular neural network fusion method of claim 2 , wherein the connecting the matching module comprises generating, by a matching interface generator, a matching interface corresponding to output parameters of the first neural network and input parameters of the second neural network.

4 . The modular neural network fusion method of claim 3 , wherein:
the matching interface comprises plurality of transform plugins, and the connecting the matching module further comprises: generating the plurality of transform plugins; and connecting the plurality of transform plugins to each other.

5 . The modular neural network fusion method of claim 3 , wherein the matching interface generator determines whether generation of a matching interface is impossible.

6 . The modular neural network fusion method of claim 3 , wherein the executing the neural network interworking service comprises generating respective input interfaces and output interfaces of the first and second neural networks.

7 . The modular neural network fusion method of claim 6 , wherein the executing the neural network interworking service further comprises:
connecting the output interface of the first neural network to the matching interface; and connecting the matching interface to the input interface of the second neural network.

8 . The modular neural network fusion method of claim 6 , wherein the executing the neural network interworking service further comprises setting activation functions contained in the input interfaces and the output interfaces of the first and second neural networks.

9 . The modular neural network fusion method of claim 1 , wherein:
the first neural network is a speech recognition neural network, and the second neural network is an image recognition neural network.

10 . A method for generating a matching interface connecting a first neural network and a second neural network in a neural network fusion apparatus, comprising:
initializing an input format of the first neural network and an output format of the second neural network; determining whether the first neural network and the second neural network are capable of being connected to each other without requiring transform plugins by comparing the input format with the output format; upon determining that the first neural network and the second neural network are not capable of being connected to each other without requiring transform plugins, determining whether there are matchable transform plugins respectively corresponding to the input format and the output format; and upon determining that there are matchable transform plugins, generating a matching interface for connecting the first neural network to the second neural network by combining the matchable transform plugins.

11 . The method of claim 10 , wherein the initializing the input format of the first neural network and the output format of the second neural network comprises:
storing, as the input format, a data type of an output parameter stored in a meta-information storage of the first neural network; and storing, as the output format, a data type of an input parameter stored in a meta-information storage of the second neural network.

12 . The method of claim 11 , wherein the initializing the input format of the first neural network and the output format of the second neural network further comprises:
storing, as the input format, a data format of reference information stored in the meta-information storage of the first neural network; and storing, as the output format, a data format of reference information stored in the meta-information storage of the second neural network.

13 . The method of claim 11 , wherein the determining whether the first neural network and the second neural network are capable of being connected to each other comprises determining whether the data type of the output parameter and the data type of the input parameter are identical to each other by comparing the data types with each other.

14 . The method of claim 11 , wherein the determining whether there are matchable transform plugins further comprises searching for transform plugins having a data type identical to that of the input format and the output format.

15 . The method of claim 11 , wherein the generating the matching interface comprises:
storing first transform plugins having a data type identical to that of the input format; and storing second transform plugins having a data type identical to that of the output format.

16 . The method of claim 15 , wherein the generating the matching interface further comprises generating a matchable combination by comparing elements of the input format with respective elements of the output format.

17 . The method of claim 15 , wherein the generating the matching interface further comprises determining whether the first transform plugins and the second transform plugins are capable of being combined with each other.

18 . A neural network fusion apparatus, comprising:
at least one processor; and a memory for storing at least one instruction that is executed by the at least one processor, wherein the at least one instruction is executed by the at least one processor so that: a first neural network is generated by a modular neural network module, a second neural network is generated by the modular neural network module, and output data of the first neural network matches input data of the second neural network through a matching interface.

19 . The neural network fusion apparatus of claim 18 , wherein:
the first neural network receives image data and outputs a text file, and the second neural network receives the text file and outputs an audio file.

20 . The neural network fusion apparatus of claim 18 , wherein:
the first neural network is a speech recognition neural network, the second neural network is an image recognition neural network, and the first neural network and the second neural network are fused through the matching interface, thus enabling a context-aware neural network to be implemented.

Assignees

• Electronics & Telecommunications Res Inst

Inventors

• Lee Kyung Hee

Classifications

• G06V10/80

  Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10) · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G10L15/16

  using artificial neural networks · CPC title