Optical convolutional neural network accelerator

Patent metadata

Field	Value
Publication number	US-11238336-B2
Application number	US-201916507854-A
Country	US
Kind code	B2
Filing date	Jul 10, 2019
Priority date	Jul 10, 2018
Publication date	Feb 1, 2022
Grant date	Feb 1, 2022

Abstract

An accelerator for modern convolutional neural networks applies the Winograd filtering algorithm in a wavelength division multiplexing integrated photonics circuit modulated by a memristor-based analog memory unit.

First claim

The invention claimed is:

1. A photonic accelerator for a convolutional neural network, comprising:
an input buffer configured to receive one or more feature maps;
a first digital-to-analog converter coupled to the input buffer, the first digital-to-analog converter configured to generate one or more analog signals corresponding to the one or more feature maps;
a coherent light source coupled to the first digital-to-analog converter, the coherent light source configured to receive the one or more analog signals from the first digital-to-analog converter and output one or more light signals based, at least in part, on the one or more analog signals;
an analog memory configured to generate one or more voltage signals;
a photonic element-wise matrix multiplication circuit coupled to the coherent light source and the analog memory, wherein the photonic element-wise matrix multiplication circuit configured to:
receive the one or more voltage signals from the analog memory;
modulate the one or more light signals responsive to the one or more voltage signals, the modulated one or more light signals having one or more wavelengths having one or more wavelength-values;
perform integration on the one or more wavelength-values of the modulated one or more light signals; and
generate an analog electrical signal corresponding to the integrated one or more wavelength-values of the modulated light signals; and

an analog-to-digital converter coupled to the photonic element-wise matrix multiplication circuit, the analog-to-digital converter configured to receive the analog electrical signal from the photonic element-wise matrix multiplication circuit, convert the analog electrical signal to a digital electrical signal, and output the digital electrical signal,
wherein the one or more voltage signals correspond to one or more filter signals.

2. The photonic accelerator of claim 1 , further comprising a photonic inverse-Winograd transform circuit, wherein:
the photonic element-wise matrix multiplication circuit is further configured to transform the one or more light signals into a Winograd domain;
the one or more filter signals are analog Winograd domain filter signals; and
the photonic inverse-Winograd transform circuit is configured to remove the one or more light signals from the Winograd domain.

3. The photonic accelerator of claim 2 , further comprising:
a filter buffer configured to receive a plurality of filters;
a Winograd transform circuit configured to transform the plurality of filters into digital-Winograd domain filter signals; and
a second digital-to-analog converter coupled to the analog memory and configured to convert the digital-Winograd domain filter signals into the analog Winograd domain filter signals,
wherein the analog memory is configured to detect the analog Winograd domain filter signals and generate the voltage signals in response thereto.

4. The photonic accelerator of claim 3 , wherein the analog memory comprises a memristor.

5. The photonic accelerator of claim 3 , wherein a first timing of the first digital-to-analog converter is controlled by a first clock and a second timing of the second digital-to-analog converter is controlled by a second clock, and the first clock runs at least ten times faster than the second clock.

6. The photonic accelerator of claim 3 , further comprising an off-chip memory and an output buffer,
wherein the input buffer is configured to receive the one or more feature maps from the off-chip memory;
the filter buffer is further configured to receive the plurality of filters from the off-chip memory; and
analog-to-digital converter is configured to output the digital electrical signal to the off-chip memory through the output buffer.

7. The photonic accelerator of claim 1 , wherein the photonic element-wise matrix multiplication circuit comprises a plurality of microring resonators and a photosensitive balanced detector summation circuit, and wherein the photonic element-wise matrix multiplication circuit is further configured to:
receive the one or more voltage signals from the analog memory and modulate the one or more light signals responsive to the one or more voltage signals using the plurality of microring resonators;
and integrate the one or more light signals so modulated using the photosensitive balanced detector summation circuit.

8. A neural network, the neural network including a system on chip, wherein the system on chip comprises:
an input buffer configured to receive one or more feature maps;
a first digital-to-analog converter coupled to the input buffer, the first digital-to-analog converter configured to generate one or more analog signals corresponding to the one or more feature maps;
a coherent light source coupled to the first digital-to-analog converter, the coherent light source configured to receive the one or more analog signals from the first digital-to-analog converter and output one or more light signals based, at least in part, on the one or more analog signals;
an analog memory configured to generate one or more voltage signals;
a photonic element-wise matrix multiplication circuit coupled to the coherent light source and the analog memory, wherein the photonic element-wise matrix multiplication circuit configured to:
receive the one or more voltage signals from the analog memory and modulate the one or more light signals responsive to the one or more voltage signals; and
generate an analog electrical signal based, at least in part, on the one or more modulated light signals; and

an analog-to-digital converter coupled to the photonic element-wise matrix multiplication circuit, the analog-to-digital converter configured to receive the analog electrical signal from the photonic element-wise matrix multiplication circuit, convert the analog electrical signal to a digital electrical signal, and output the digital electrical signal,
wherein the one or more voltage signals correspond to one or more filter signals.

9. The neural network of claim 8 , wherein the photonic element-wise matrix multiplication circuit comprises a plurality of microring resonators and a photosensitive balanced detector summation circuit, and wherein the photonic element-wise matrix multiplication circuit is further configured to:
receive the one or more voltage signals from the analog memory and modulate the one or more light signals responsive to the one or more voltage signals using the plurality of microring resonators;
and integrate the one or more light signals so modulated using the photosensitive balanced detector summation circuit.

10. The neural network of claim 9 , wherein the system on chip further comprises a photonic inverse-Winograd transform circuit, wherein:
the photonic element-wise matrix multiplication circuit is further configured to transform the one or more light signals into a Winograd domain;
the one or more filter signals are analog Winograd domain filter signals; and
the photonic inverse-Winograd transform circuit is configured to remove the one or more light signals from the Winograd domain.

11. The neural network of claim 10 , wherein the system on chip further comprises:
a filter buffer configured to receive a plurality of filters;
a Winograd transform circuit configured to transform the plurality of filters into digital-Winograd domain filter signals; and
a second digital-to-analog converter coupled to the analog memory and configured to convert the digital-Winograd domain filter signals into the analog Winograd domain filter signals,
wherein the analog memory is configured to detect the analog Winograd domain filter signals and generate the voltage signals in response there

Assignees

• Univ George Washington

Inventors

• Mehrabian Armin
• Sorger Volker J
• El-Ghazawi Tarek
• Miscuglio Mario

Classifications

• G06N3/067 Primary

  using optical means · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F1/06

  Clock generators producing several clock signals {(G06F1/08 - G06F1/14 take precedence)} · CPC title

• G06N3/0675 Primary

  using electro-optical, acousto-optical or opto-electronic means · CPC title