Photonic in-memory co-processor for convolutional operations

Patent metadata

Field	Value
Publication number	US-2022012013-A1
Application number	US-202016925998-A
Country	US
Kind code	A1
Filing date	Jul 10, 2020
Priority date	Jul 10, 2020
Publication date	Jan 13, 2022
Grant date	—

Abstract

A co-processor for performing a matrix multiplication of an input matrix with a data matrix in one step may be provided. The co-processor receives input signals for the input matrix as optical signals. A plurality of photonic memory elements is arranged at crossing points of an optical waveguide crossbar array. The plurality of memory elements is configured to store values of the data matrix. Input signals are connected to input lines of the optical waveguide crossbar array. Output lines of the optical waveguide crossbar array represent a dot-product between a respective column of the optical waveguide crossbar array and the received input signals, and values of elements of the input matrix to be multiplied with the data matrix correspond to light intensities received at input lines of the respective photonic memory elements. Additionally, different wavelengths are used for each column of the input matrix optical signals.

First claim

1 . A co-processor for performing a matrix multiplication of an input matrix with a data matrix in one step, said co-processor comprising:
a receiving unit adapted for receiving input signals for said input matrix as optical signals; a plurality of photonic memory elements, arranged at crossing points of an optical waveguide crossbar array, wherein said plurality of photonic memory elements is configured to store values of said data matrix, and wherein said receiving unit is connected to input lines of said optical waveguide crossbar array; wherein a respective output signal of each of output lines of said optical waveguide crossbar array represents a dot-product between a respective column of said optical waveguide crossbar array and said received input signals; and wherein values of elements of said input matrix to be multiplied with said data matrix correspond to light intensities received at input lines of said respective photonic memory elements; and wherein different wavelengths are used for each column of said input matrix optical signals, such that said input matrix is multiplied with said data matrix in one step.

2 . The co-processor according to claim 1 , wherein different wavelengths are used for each element value of said input matrix.

3 . The co-processor according to claim 1 , wherein different wavelengths are multiplexed before said optical waveguide crossbar array and de-multiplexed after said optical waveguide crossbar array.

4 . The co-processor according to claim 1 , wherein each of said plurality of photonic memory elements comprises, adjacent to a nano-photonic waveguide, a phase-change material layer.

5 . The co-processor according to claim 1 , wherein said elements of said data matrix correspond to values of elements of linearized convolutional kernels of a convolutional neural network.

6 . The co-processor according to claim 5 , wherein said signal values of said output lines of said optical waveguide crossbar array correspond to elements of a convolution result.

7 . The co-processor according to claim 1 , wherein each of said output lines of said optical waveguide crossbar array is connected to a respective photodetector.

8 . The co-processor according to claim 1 , wherein each of said output lines of said optical waveguide crossbar array is connected to a respective integrated optical-digital signal processing unit.

9 . The co-processor according to claim 1 , wherein one of said columns of said optical waveguide crossbar array is left in a reference state.

10 . The co-processor according to claim 1 , wherein said optical waveguide crossbar array comprises, at each crossing point, one of said photonic memory elements connecting a respective one of said input lines with a respective one of said output lines.

11 . A computer-implemented method for operating a co-processor for performing a matrix multiplication of an input matrix with a data matrix in one step, wherein a plurality of photonic memory elements is arranged at crossing points of an optical waveguide crossbar array, and wherein said plurality of photonic memory elements is configured to store values of said data matrix, and wherein said receiving unit is connected to input lines of said optical waveguide crossbar array, said method comprising:
receiving input signals for said input matrix as optical signals; wherein a respective output signal of each of output lines of said optical waveguide crossbar array represents a dot-product between a respective column of said optical waveguide crossbar array and said received input signals; and wherein values of elements of said input matrix to be multiplied with said data matrix correspond to light intensities received at input lines of said respective photonic memory elements; and wherein different wavelength are used for each column of said input matrix optical signals, such that said input matrix is multiplied with said data matrix in one step.

12 . The method according to claim 11 , also comprising using different wavelengths for each element value of said input matrix.

13 . The method according to claim 11 , wherein different wavelengths are multiplexed before said optical waveguide crossbar array and de-multiplexed after said optical waveguide crossbar array.

14 . The method according to claim 11 , wherein each of said plurality of photonic memory elements comprises, adjacent to a nano-photonic waveguide, a phase-change material layer.

15 . The method according to claim 11 , wherein said elements of said data matrix correspond to values of elements of linearized convolutional kernels of a convolutional neural network.

16 . The method according to claim 15 , wherein said signal values of said output lines of said optical waveguide crossbar array correspond to elements of a convolutional result.

17 . The co-processor according to claim 11 , wherein each of said output lines of said optical waveguide crossbar array is connected to a respective photodetector.

18 . The method according to claim 11 , also comprising:
maintaining one of said columns of said optical waveguide crossbar array in a reference state.

19 . The method according to claim 112 , wherein said optical waveguide crossbar array comprises, at each crossing point, one of said photonic memory elements connecting a respective one of said input lines with a respective one of said output lines.

20 . A computer program product for operating a co-processor for performing a matrix multiplication of an input matrix with a data matrix in one step, wherein a plurality of photonic memory elements is arranged at crossing points of an optical waveguide crossbar array, wherein said plurality of photonic memory elements is configured to store values of said data matrix, and wherein said receiving unit is connected to input lines of said optical waveguide crossbar array;
the computer program product comprising a computer readable storage medium having program instructions embodied therewith, said program instructions being executable by one or more computing systems or controllers to cause said one or more computing systems to: control a reception of input signals for said input matrix as optical signals, wherein a respective output signal of each of output lines of said optical waveguide crossbar array represents a dot-product between a respective column of said optical waveguide crossbar array and said received input signals; and wherein values of elements of said input matrix to be multiplied with said data matrix correspond to light intensities received at input lines of said respective photonic memory elements; and wherein different wavelength are used for each column of said input matrix optical signals, such that said input matrix is multiplied with said data matrix in one step.

Assignees

• IBM
• Univ Of Muenster
• Univ Exeter
• Univ Oxford Innovation Ltd

Inventors

• Sebastian Abu
• Le Gallo-Bourdeau Manuel
• Wright Christopher David
• Youngblood Nathan
• Bhaskaran Harish
• Li Xuan
• Pernice Wolfram
• Feldmann Johannes

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06E3/003

  forming integrals of products, e.g. Fourier integrals, Laplace integrals, correlation integrals; for analysis or synthesis of functions using orthogonal functions · CPC title

• G06F17/16

  Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} · CPC title

• G06E3/008

  Matrix or vector computation · CPC title