Imaging device utilizing a neural network

Patent metadata

Field	Value
Publication number	US-12462142-B2
Application number	US-202017629499-A
Country	US
Kind code	B2
Filing date	Jul 30, 2020
Priority date	Aug 9, 2019
Publication date	Nov 4, 2025
Grant date	Nov 4, 2025

Abstract

A system with high processing speed and low power consumption is provided. The system includes an imaging device and an arithmetic circuit. The imaging device includes an imaging portion, a first memory portion, and an arithmetic portion, and the arithmetic circuit includes a second memory portion. The imaging portion has a function of converting light reflected by an external subject into image data, and the first memory portion has a function of storing the image data and a first filter for performing first convolutional processing in a first layer of a neural network. The arithmetic portion has a function of performing the first convolutional processing using the image data and the first filter to generate first data. The second memory portion has a function of storing the first data and a plurality of filters. The arithmetic circuit has a function of generating a depth map of the image data.

First claim

The invention claimed is:

1 . A system comprising:
an imaging device and an arithmetic circuit, wherein the imaging device comprises an imaging portion, a first memory portion, and an arithmetic portion, wherein the arithmetic circuit comprises a second memory portion, wherein the imaging portion is configured to convert light reflected by an external object into image data, wherein the first memory portion is configured to store the image data, a first filter for performing first convolutional processing in a first layer of a first neural network, and a second filter for performing second convolutional processing in a first layer of a second neural network, wherein the arithmetic portion is configured to perform the first convolutional processing on the image data using the first filter as multiplier data and a partial region of the image data as multiplicand data to generate first data, wherein the arithmetic portion is configured to perform the second convolutional processing on the image data using the second filter to generate second data, wherein the second memory portion is configured to store the first data, the second data, and a plurality of filters for performing convolutional processing in a second layer and subsequent layers of the first neural network and in a fourth layer and subsequent layers of the second neural network, wherein the arithmetic circuit is configured to perform processing in the second layer and the subsequent layers of the first neural network using the first data to output third data from an output layer of the first neural network, wherein the arithmetic circuit is configured to perform pooling processing on the second data in the second layer of the second neural network to generate fourth data, wherein the arithmetic circuit is configured to combine the third data and the fourth data in a third layer of the second neural network to generate fifth data, and wherein the arithmetic circuit is configured to perform processing in the fourth layer and subsequent layers of the second neural network using the fifth data to output a depth map of the image data from an output layer of the second neural network; wherein the arithmetic circuit is configured to determine an image using the image data obtained through the convolutional processing and the pooling processing in a fully connected layer of the second neural network, wherein the fully connected layer has a structure in which all nodes in one layer are connected to all nodes in a subsequent layer, and wherein the arithmetic circuit is configured to generate a three-dimensional image using the image data and the depth map.

2 . The system according to claim 1 ,
further comprising a memory device, wherein the memory device is configured to store the first filter and the plurality of filters, wherein the memory device is configured to transmit the first filter to the first memory portion, and wherein the memory device is configured to transmit the plurality of filters to the second memory portion.

3 . A system comprising:
an imaging device comprising a transistor including an oxide semiconductor layer which is configured to function as an active layer, an arithmetic circuit, and a memory device, wherein the imaging device comprises an imaging portion, a first memory portion, and an arithmetic portion, wherein the arithmetic circuit comprises a second memory portion, wherein the imaging portion is configured to convert light reflected by an external object into image data, wherein the first memory portion is configured to store the image data, a first filter for performing first convolutional processing in a first layer of a first neural network, and a second filter for performing second convolutional processing in a first layer of a second neural network, wherein the arithmetic portion is configured to perform the first convolutional processing on the image data using the first filter to generate first data, wherein the arithmetic portion is configured to perform the second convolutional processing on the image data using the second filter to generate second data, wherein the second memory portion is configured to store the first data, the second data, and a plurality of filters for performing convolutional processing in a second layer and subsequent layers of the first neural network and convolutional processing in a fourth layer and subsequent layers of the second neural network, wherein the arithmetic circuit is configured to perform processing in the second layer and the subsequent layers of the first neural network using the first data to output third data from an output layer of the first neural network, wherein the arithmetic circuit is configured to perform pooling processing on the second data in the second layer of the second neural network to generate fourth data, wherein the arithmetic circuit is configured to combine the third data and the fourth data in a third layer of the second neural network to generate fifth data, wherein the arithmetic circuit is configured to perform processing in the fourth layer and subsequent layers of the second neural network using the fifth data to output a depth map of the image data from an output layer of the second neural network, wherein the arithmetic circuit is configured to determine an image using the image data obtained through the convolutional processing and the pooling processing in a fully connected layer of the second neural network, and wherein the fully connected layer has a structure in which all nodes in one layer are connected to all nodes in a subsequent layer; wherein the arithmetic circuit is configured to generate a three-dimensional image using the image data and the depth map.

4 . The system according to claim 3 ,
wherein the memory device is configured to store the first filter, the second filter, and the plurality of filters, wherein the memory device is configured to transmit the first filter and the second filter to the first memory portion, and wherein the memory device is configured to transmit the plurality of filters to the second memory portion.

5 . The system according to claim 3 , wherein the image data output from a pooling layer of the second neural network is a two-dimensional feature map and is unfolded into a one-dimensional feature map when input into the fully connected layer of the second neural network.

Assignees

• Semiconductor Energy Lab

Inventors

• Koumura Yusuke
• Inoue Koki
• Kimotsuki Ayana
• Nagashima Fumiya

Classifications

• G06T1/60

  Memory management · CPC title

• G06N3/063

  using electronic means · CPC title

• G06T7/50 Primary

  Depth or shape recovery · CPC title

• H10D84/00

  Integrated devices formed in or on semiconductor substrates that comprise only semiconducting layers, e.g. on Si wafers or on GaAs-on-Si wafers · CPC title

• H10D84/0126

  the components including insulated gates, e.g. IGFETs · CPC title