Back propagation gates and storage capacitor for neural networks

Patent metadata

Field	Value
Publication number	US-9779355-B1
Application number	US-201615266332-A
Country	US
Kind code	B1
Filing date	Sep 15, 2016
Priority date	Sep 15, 2016
Publication date	Oct 3, 2017
Grant date	Oct 3, 2017

Abstract

Technical solutions are described for implementing a neural network. An example system includes a crosspoint array including a plurality of nodes, each node representing a weight assigned to a neuron of the neural network. The system also includes a capacitor associated with a set of nodes from the plurality of nodes, where the capacitor is configured to store a current value corresponding to a sum of outputs from each respective node from the set of nodes. The system also includes a clocking circuit that initiates a forward pass to propagate the current value stored in the capacitor to a subsequent layer of the neural network. The clocking circuit further initiates a backward pass to propagate the current value stored in the capacitor to a preceding layer of the neural network. The clocking circuit further initiates a weight-update pass to update the weights in the neural network.

First claim

What is claimed is:

1. A system for implementing a neural network, the system comprising:
a crosspoint array comprising a plurality of nodes, each node representing a weight assigned to a neuron of the neural network;
a capacitor associated with a set of nodes from the plurality of nodes, wherein the capacitor is configured to store a current value corresponding to a sum of outputs from each respective node from the set of nodes; and

a clocking circuit comprises a first clocking device, a second clocking device, a third clocking device, and a fourth clocking device, wherein the clocking circuit is configured to:
initiate a forward pass to propagate the current value stored in the capacitor to a subsequent layer of the neural network by setting the second clocking device and the third clocking device;
initiate a backward pass to propagate the current value stored in the capacitor to a preceding layer of the neural network by setting the first clocking device and the third clocking device; and
initiate a weight-update pass to update the weights in the neural network by setting the second clocking device and the fourth clocking device.

2. The system of claim 1 , wherein the capacitor stores the sum of outputs from each respective node from the set of nodes in response to a forward pass initiation.

3. The system of claim 1 , wherein the capacitor propagates the stored current value to the preceding layer of the neural network in response to the backward pass initiation.

4. The system of claim 1 , wherein the weights of the neural network are updated based on the value stored in the capacitor and an error signal received from the subsequent layer of the neural network.

5. The system of claim 1 , wherein the crosspoint array comprises:
a set of conductive row wires;
a set of conductive column wires configured to form a plurality of crosspoints at intersections between the set of conductive row wires and the set of conductive column wires; and
a two-terminal RPU at each of the plurality of crosspoints.

6. The system of claim 5 , wherein the two-terminal RPU comprises:
a first terminal;
a second terminal; and
an active region having a conduction state;
wherein the active region is configured to effect a non-linear change in the conduction state based on at least one first encoded signal applied to the first terminal and at least one second encoded signal applied to the second terminal.

7. A crosspoint array for implementing a neural network, the crosspoint array comprising:
a set of conductive row wires;
a set of conductive column wires configured to form a plurality of crosspoints at intersections between the set of conductive row wires and the set of conductive column wires;
a memristor at each of the plurality of crosspoints configured to store a weight of the neural network;
a capacitor associated with a set of crosspoints from the crosspoint array, wherein the capacitor is configured to store a current value corresponding to a sum of outputs from each respective memristor from the set of crosspoints; and
a clocking circuit, which comprises a first clocking device, a second clocking device, a third clocking device, and a fourth clocking device, wherein the clocking circuit is configured to:
initiate a forward pass to propagate the current value stored in the capacitor to a subsequent layer of the neural network by setting the second clocking device and the third clocking device;
initiate a backward pass to propagate the current value stored in the capacitor to a preceding layer of the neural network by setting the first clocking device and the third clocking device; and
initiate a weight-update pass to update the weights in the neural network by setting the second clocking device and the fourth clocking device.

8. The crosspoint array of claim 7 , wherein the memristor comprises a two-terminal resistive processing unit (RPU).

9. The crosspoint array of claim 8 , the two-terminal RPU comprises:
a first terminal;
a second terminal; and
an active region having a conduction state;
wherein the active region is configured to effect a non-linear change in the conduction state based on at least one first encoded signal applied to the first terminal and at least one second encoded signal applied to the second terminal.

10. A non-transitory computer program product for implementing a neural network, the computer program product comprising a computer readable storage medium, the computer readable storage medium comprising computer executable instructions, wherein the computer readable storage medium comprises instructions to:
set one or more clocking devices from a clocking circuitry of a crosspoint array,
wherein the crosspoint array comprises:
a set of conductive row wires;
a set of conductive column wires configured to form a plurality of crosspoints at intersections between the set of conductive row wires and the set of conductive column wires;
a plurality of memristors, wherein a memristor is at each of the plurality of crosspoints, and the memristor configured to store a weight of the neural network; and
a plurality of capacitors, wherein a capacitor is associated with a corresponding set of crosspoints from the crosspoint array; and

the clocking circuitry comprises a first clocking device, a second clocking device, a third clocking device, and a fourth clocking device, and wherein executing the instructions causes the clocking circuitry to:
initiate a forward pass that comprises storing, in the capacitor, a current value corresponding to a sum of outputs from each respective memristor from the corresponding set of crosspoints, and propagating current values stored in the capacitor to a subsequent layer of the neural network by setting the second clocking device and the third clocking device.

11. The non-transitory computer program product of claim 10 , wherein the computer readable storage medium further comprises instructions to initiate a backward pass to propagate the current value stored in the capacitor to a preceding layer of the neural network by setting the first clocking device and the third clocking device.

12. The non-transitory computer program product of claim 10 , wherein the computer readable storage medium further comprises instructions to initiate a weight-update pass to update the weights in the memristors by setting the second clocking device and the fourth clocking device.

Assignees

• IBM

Inventors

• Leobandung Effendi

Classifications

• G06N3/065

  Analogue means · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title