Visual programming for deep learning

Patent metadata

Field	Value
Publication number	US-12079600-B2
Application number	US-202017615080-A
Country	US
Kind code	B2
Filing date	May 6, 2020
Priority date	Jun 28, 2019
Publication date	Sep 3, 2024
Grant date	Sep 3, 2024

Abstract

Implementations of the present disclosure relate to visual programming for deep learning. A computer-implemented method comprises presenting a visual representation of an artificial neural network, the visual representation comprising graphical elements representing layers of the artificial neural network; in response to receiving a drag-and-drop operation on the graphical elements, modifying an intermediate representation of the artificial neural network, wherein the intermediate representation is independent of a deep learning framework and the drag-and-drop operation is configured to modify connections between the graphical elements; and modifying, based on the intermediate representation of the artificial neural network, code of the artificial neural network for a target deep learning framework.

First claim

The invention claimed is:

1. A computer-implemented method, comprising:
presenting a visual representation of an artificial neural network, wherein the visual representation of the artificial neural network includes graphical elements representing layers of the artificial neural network;
in response to receiving a drag-and-drop operation on the graphical elements, modifying an intermediate representation of the artificial neural network, wherein the intermediate representation of the artificial neural network is independent of a deep learning framework and the drag-and-drop operation is configured to modify connections between the graphical elements including automatically connecting a first graphical element to a second graphical element of the graphical elements;
modifying, based on the intermediate representation of the artificial neural network, code of the artificial neural network for a target deep learning framework; and
in response to receiving an instruction for changing the target deep learning framework to a further target deep learning framework, determining code of the artificial neural network for the further target deep learning framework based on the intermediate representation of the artificial neural network.

2. The computer-implemented method of claim 1 , further comprising:
in response to an editing operation on the code of the artificial neural network for the target deep learning framework, modifying the intermediate representation of the artificial neural network; and
adjusting the visual representation of the artificial neural network based on the modified intermediate representation of the artificial neural network.

3. The computer-implemented method of claim 1 , further comprising:
in response to receiving the drag-and-drop operation on the graphical elements, validating dimensions of data associated with the layers of the artificial neural network.

4. The computer-implemented method of claim 1 , further comprising:
in response to receiving a search operation associated with a keyword, presenting graphical elements representing at least one candidate layer corresponding to the keyword; and
in response to receiving a selection of graphical elements of the at least one candidate layer, adding the selected graphical elements of the at least one candidate layer to the visual representation of the artificial neural network.

5. The computer-implemented method of claim 1 , further comprising:
presenting code stubs for customizing metrics of the artificial neural network; and
in response to an editing operation on the code stubs, customizing the metrics of the artificial neural network.

6. The computer-implemented method of claim 1 , further comprising:
modifying the intermediate representation of the artificial neural network in response to at least one of:
adding, into the visual representation of the artificial neural network, a new graphical element representing a layer of the artificial neural network;
deleting, from the visual representation of the artificial neural network, a graphical element representing a layer of the artificial neural network; and
modifying parameters of a graphical element representing a layer of the artificial neural network.

7. A device comprising:
a processing unit; and
a memory coupled to the processing unit and having instructions stored thereon, the instructions, when executed by the processing unit, causing the device to perform acts comprising:
presenting a visual representation of an artificial neural network, wherein the visual representation of the artificial neural network includes graphical elements representing layers of the artificial neural network;
in response to receiving a drag-and-drop operation on the graphical elements, modifying an intermediate representation of the artificial neural network, wherein the intermediate representation of the artificial neural network is independent of a deep learning framework and the drag-and-drop operation is configured to modify connections between the graphical elements including automatically connecting a first graphical element to a second graphical element of the graphical elements;
modifying, based on the intermediate representation of the artificial neural network, code of the artificial neural network for a target deep learning framework; and
in response to receiving an instruction for changing the target deep learning framework to a further target deep learning framework, determining code of the artificial neural network for the further target deep learning framework based on the intermediate representation of the artificial neural network.

8. The device of claim 7 , wherein the acts further comprise:
in response to an editing operation on the code of the artificial neural network for the target deep learning framework, modifying the intermediate representation of the artificial neural network; and
adjusting the visual representation of the artificial neural network based on the modified intermediate representation of the artificial neural network.

9. The device of claim 7 , wherein the acts further comprise:
in response to receiving the drag-and-drop operation on the graphical elements, validating dimensions of data associated with the layers of the artificial neural network.

10. The device of claim 7 , wherein the acts further comprise:
in response to receiving a search operation associated with a keyword, presenting graphical elements representing at least one candidate layer corresponding to the keyword; and
in response to receiving a selection of graphical elements of the at least one candidate layer, adding the selected graphical elements of the at least one candidate layer to the visual representation of the artificial neural network.

11. The device of claim 7 , wherein the acts further comprise:
presenting code stubs for customizing metrics of the artificial neural network; and
in response to an editing operation on the code stubs, customizing the metrics of the artificial neural network.

12. The device of claim 7 , wherein the acts further comprise:
modifying the intermediate representation of the artificial neural network in response to at least one of:
adding, into the visual representation of the artificial neural network, a new graphical element representing a layer of the artificial neural network;
deleting, from the visual representation of the artificial neural network, a graphical element representing a layer of the artificial neural network; and
modifying parameters of a graphical element representing a layer of the artificial neural network.

13. A computer program product stored in a non-transitory computer storage medium and comprising computer-executable instructions which, when executed by a device, cause the device to perform acts comprising:
presenting a visual representation of an artificial neural network, wherein the visual representation of the artificial neural network includes graphical elements representing layers of the artificial neural network;
in response to receiving a drag-and-drop operation on the graphical elements, modifying an intermediate representation of the artificial neural network, wherein the intermediate representation of the artificial neural network is independent of a deep learning framework and the drag-and-drop operation is configured to modify connections between the graphical elements including automatically connecting a first graphical element to a second graphical element of the graphical elements;
modifying, based on the intermediate representation of the artificial neural network, code of the artificial neural network for a target deep learn

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Lin Haoxiang
• Yang Mao
• Liu Shuguang
• Chen Cheng

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/048

  Activation functions · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/063

  using electronic means · CPC title