High density parallel proximal image processing

Patent metadata

Field	Value
Publication number	US-11330215-B2
Application number	US-202016953554-A
Country	US
Kind code	B2
Filing date	Nov 20, 2020
Priority date	Mar 25, 2018
Publication date	May 10, 2022
Grant date	May 10, 2022

Abstract

A distributed, parallel, image capture and processing architecture provides significant advantages over prior art systems. A very large array of computational circuits—in some embodiments, matching the size of the pixel array—is distributed around, within, or beneath the pixel array of an image sensor. Each computational circuit is dedicated to, and in some embodiments is physically proximal to, one, two, or more associated pixels. Each computational circuit is operative to perform computations on one, two, or more pixel values generated by its associated pixels. The computational circuits all perform the same operation(s), in parallel. In this manner, a very large number of pixel-level operations are performed in parallel, physically and electrically near the pixels. This obviates the need to transfer very large amounts of pixel data from a pixel array to a CPU/memory, thus alleviating the significant high-speed performance constraints placed on modern image sensors.

First claim

What is claimed is:

1. An imaging circuit, comprising:
an array of pixel sensors, each pixel sensor operative to generate a pixel value in response to incident photons, said array of pixel sensors arranged in rows and columns;
a plurality of Analog to Digital Converters (ADC) disposed adjacent said pixel sensor array along at least one side of one or both of said rows and columns and configured to digitize pixel values; and
a plurality of computational circuits operating in parallel and equal in number to the rows or columns, each computational circuit being disposed proximate to said ADCs and configured to perform one or more computations on one, two, or more digital pixel values, wherein each computational circuit, other than those at either end, receives input from an associated row or column and both immediately adjacent rows or columns.

2. The imaging circuit of claim 1 wherein said plurality of computational circuits comprise a plurality of combinatorial, sequential, or arithmetic computational circuits.

3. The imaging circuit of claim 2 further comprising memory associated with each row or column and operative to store at least three successive pixel values as the pixel values are shifted out of the row or column.

4. The imaging circuit of claim 3 wherein each combinatorial, sequential, or arithmetic computational circuit, other than those at either end, receives as input the current and previous two pixel values for the associated row or column, and both immediately adjacent rows or columns.

5. The imaging circuit of claim 2 wherein each combinatorial, sequential, or arithmetic computational circuit is an Arithmetic Logic Unit (ALU).

6. The imaging circuit of claim 2 wherein each combinatorial, sequential, or arithmetic computational circuit is implemented as a programmable logic block.

7. The imaging circuit of claim 1 wherein said one, two, or more pixel values comprise a past and a current value generated by the same pixel.

8. The imaging circuit of claim 1 wherein said incident photons comprise visible light.

9. The imaging circuit of claim 1 wherein said incident photons comprise ultraviolet light.

10. The imaging circuit of claim 1 wherein said incident photons comprise near infrared light.

11. The imaging circuit of claim 1 wherein said incident photons comprise thermal infrared radiation.

12. The imaging circuit of claim 1 wherein said one or more computations relate to an algorithm selected from the group consisting of a stretch operation, a scalar multiply, add, subtract or divide operation, optical flow, Sobel edge detection, Difference of Gaussians gradient detection, histogram of gradients, Canny Corners, H.264/265 encoding, JPEG encoding, demosaic, debayer, motion detection using Gaussian Mixture Model, n-frame differencing, differential encoding, jitter detection/suppression, vibration detection/suppression, flicker detection/suppression, LiFi demodulation/decode, digital infinite impulse response filtering, digital finite impulse response filtering, Goertzel filtering, temporal FFT, spatial FFT, local area median, 1D or 2D Gaussian blur, Unsharp Mask, 1D or 2D box filter, running average, regional running average, and regional running average of differences.

13. The imaging circuit of claim 1 wherein a group comprises an n×m zone of pixel sensors, where n and m are integers, and wherein performing one or more computations on two or more pixel values comprises downsampling the n×m zone of pixel values to one pixel value.

14. The imaging circuit of claim 13 wherein each group of pixels comprises a square region of pixel sensors.

15. The imaging circuit of claim 1 further comprising:
a plurality of light sources; and
a controller connected to the light sources and the arrays of pixel sensors and computational circuits; and operative to receive image-processed information from the outputs of the computational circuits, and further operative to control the plurality of light sources in response to the image-processed information.

16. The imaging circuit of claim 15 wherein the controller is further connected to a network of two or more imaging circuits including light sources.

17. An imaging circuit, comprising:
an array of pixel sensors, each pixel sensor operative to generate a pixel value in response to incident photons, said array of pixel sensors arranged in rows and columns;
a plurality of Analog to Digital Converters (ADC), each associated with a row or column, and configured to digitize pixel values; and
a plurality of computational circuits operating in parallel and equal in number to the rows or columns, each computational circuit being associated with an ADC and configured to perform one or more computations on one, two, or more digital pixel values, wherein each computational circuit, other than those at either end, receives input from an associated row or column and both immediately adjacent rows or columns.

18. A control system, comprising:
a plurality of imaging circuits, each imaging circuit comprising
an array of pixel sensors, each pixel sensor operative to generate a pixel value in response to incident photons, said array of pixel sensors arranged in rows and columns;
a plurality of Analog to Digital Converters (ADC) disposed adjacent said pixel sensor array along at least one side of one or both of said rows and columns and configured to digitize pixel values; and
a plurality of computational circuits operating in parallel and equal in number to the rows or columns, each computational circuit being disposed proximate to said ADCs and configured to perform one or more computations on one, two, or more digital pixel values, wherein each computational circuit, other than those at either end, receives input from an associated row or column and both immediately adjacent rows or columns; and

a controller operative to receive and further process image-processed information from each imaging circuit, and operative to control one or more devices in response to the information.

19. The control system of claim 18 wherein the one or more device controlled by said controller comprise lighting fixtures.

20. The control system of claim 19 wherein the lighting fixtures include the imaging circuits.

21. The control system of claim 18 wherein the one or more device controlled by said controller comprise HVAC systems of one or more buildings or structures.

22. The control system of claim 18 wherein the one or more device controlled by said controller comprise access control devices.

23. The control system of claim 18 wherein the one or more device controlled by said controller comprise package handling devices.

Assignees

• Ideal Ind Lighting Llc

Inventors

• Roberts John
• Bowser Robert

Classifications

• G05B15/02 Primary

  electric · CPC title

• H04N25/77

  Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components · CPC title

• H04N25/713

  Transfer or readout registers; Split readout registers or multiple readout registers · CPC title

• H04N25/767

  Horizontal readout lines, multiplexers or registers · CPC title

• H04N25/78

  Readout circuits for addressed sensors, e.g. output amplifiers or A/D converters · CPC title