Interpretation method for neural network model, electronic device and storage medium

Patent metadata

Field	Value
Publication number	US-2023093528-A1
Application number	US-202218070775-A
Country	US
Kind code	A1
Filing date	Nov 29, 2022
Priority date	Jan 19, 2022
Publication date	Mar 23, 2023
Grant date	—

Abstract

An interpretation method for a neural network model is provided. Input data and output data corresponding to the input data of a neural network model are acquired, in which the neural network model includes layers of networks connected sequentially, and each layer of network corresponds to a plurality of candidate concepts. A key inference path through which the output data is obtained by the neural network model based on the input data are acquired, in which the key inference path includes target concepts respectively used by the layers of networks when the input data is processed in the neural network model, in which the target concepts are selected from the plurality of candidate concepts. Interpretation information corresponding to the layers of networks is determined according to the target concepts corresponding to the layers of networks, respectively. The key inference path and the interpretation information are output.

First claim

What is claimed is:

1 . An interpretation method for a neural network model, comprising:
acquiring input data and output data corresponding to the input data of a neural network model, wherein the neural network model comprises layers of networks connected sequentially, and each layer of network corresponds to a plurality of candidate concepts; acquiring a key inference path through which the output data is obtained by the neural network model based on the input data, wherein the key inference path comprises target concepts respectively used by the layers of networks when the input data is processed in the neural network model, wherein the target concepts are selected from the plurality of candidate concepts; determining interpretation information corresponding to the layers of networks according to the target concepts corresponding to the layers of networks, respectively; and outputting the key inference path and the interpretation information.

2 . The method of claim 1 , wherein acquiring the key inference path through which the output data is obtained by the neural network model based on the input data comprises:
acquiring a j th layer of network corresponding to the output data, wherein j is equal to N, and N is a total number of layers of networks in the neural network model; acquiring a target concept in the i th layer of network; acquiring quantitative relationships between candidate concepts in an i th layer of network and the target concept, respectively, wherein i is equal to j minus 1; determining a target concept in the layer of network according to the candidate concepts in the i th layer of network and the quantitative relationships; subtracting 1 from j, and executing acquiring the target concept in the j th layer of network when j is greater than 2; and generating the key inference path according to the target concepts in the layers of networks when j is equal to 2.

3 . The method of claim 2 , wherein determining the target concepts of the i th layer of network according to the candidate concepts in the i th layer of network and the quantitative relationships comprises:
acquiring importance values of the quantitative relationships; ranking the quantitative relationships in a descending order of the importance values of the quantitative relationships to obtain a ranking result; taking out the quantitative relationships sequentially according to the ranking result, and acquiring candidate concepts corresponding to the quantitative relationships from the candidate concepts in the i th layer of network; accumulating estimated values of the candidate concepts corresponding to the quantitative relationships taken out until an accumulated value is greater than a preset threshold; and determining the target concept in the i th layer of network from the candidate concepts corresponding to the quantitative relationships taken out from the ranking result.

4 . The method of claim 1 , wherein the interpretation information comprises semantic information of the target concept.

5 . The method of claim 4 , wherein the interpretation information further comprises sample characteristics of a target sample corresponding to the target concept.

6 . The method of claim 1 , further comprising:
acquiring a quantitative relationship between target concepts in two adjacent layers of networks according to the target concepts in the two adjacent layers of networks for any two adjacent layers of networks in the key inference path; and marking the quantitative relationship between the target concepts in the two adjacent layers of networks in the key inference path.

7 . An electronic device, comprising:
at least one processor; and a memory communicatively connected with the at least one processor for storing instructions executable by the at least one processor; wherein the at least one processor is configured to: acquire input data and output data corresponding to the input data of a neural network model, wherein the neural network model comprises layers of networks connected sequentially, and each layer of network corresponds to a plurality of candidate concepts; acquire a key inference path through which the output data is obtained by the neural network model based on the input data, wherein the key inference path comprises target concepts respectively used by the layers of networks when the input data is processed in the neural network model, wherein the target concepts are selected from the plurality of candidate concepts; determine interpretation information corresponding to the layers of networks according to the target concepts corresponding to the layers of networks, respectively; and output the key inference path and the interpretation information.

8 . The electronic device of claim 7 , wherein the at least one processor is configured to:
acquire a j th layer of network corresponding to the output data, wherein j is equal to N, and N is a total number of layers of networks in the neural network model; acquire a target concept in the j th layer of network; acquire quantitative relationships between candidate concepts in an i th layer of network and the target concept, respectively, wherein i is equal to j minus 1; determine a target concept in the i th layer of network according to the candidate concepts in the layer of network and the quantitative relationships; subtract 1 from j, and execute acquiring the target concept in the j th layer of network when j is greater than 2; and generate the key inference path according to the target concepts in the layers of networks when j is equal to 2.

9 . The electronic device of claim 8 , wherein the at least one processor is configured to:
acquire importance values of the quantitative relationships; rank the quantitative relationships in a descending order of the importance values of the quantitative relationships to obtain a ranking result; take out the quantitative relationships sequentially according to the ranking result, and acquiring candidate concepts corresponding to the quantitative relationships from the candidate concepts in the layer of network; accumulate estimated values of the candidate concepts corresponding to the quantitative relationships taken out until an accumulated value is greater than a preset threshold; and determine the target concept in the layer of network from the candidate concepts corresponding to the quantitative relationships taken out from the ranking result.

10 . The electronic device of claim 7 , wherein the interpretation information comprises semantic information of the target concept.

11 . The electronic device of claim 10 , wherein the interpretation information further comprises sample characteristics of a target sample corresponding to the target concept.

12 . The electronic device of claim 7 , wherein the at least one processor is configured to:
acquire a quantitative relationship between target concepts in two adjacent layers of networks according to the target concepts in the two adjacent layers of networks for any two adjacent layers of networks in the key inference path; and mark the quantitative relationship between the target concepts in the two adjacent layers of networks in the key inference path.

13 . A non-transitory computer-readable storage medium having stored therein computer instructions that, when executed by a computer, cause the computer to perform:
acquiring input data and output data corresponding to the input data of a neural network model, wherein the neural network model comprises layers of networks connected sequentially, and each layer of network corresponds to a pl

Assignees

• Beijing Baidu Netcom Sci & Tech Co Ltd

Inventors

• Sun Ying
• Zhu Hengshu
• Qin Chuan
• Zhuang Fuzhen
• Xiong Hui

Classifications

• G06N3/044 Primary

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/04 Primary

  Architecture, e.g. interconnection topology · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06N3/08

  Learning methods · CPC title