CMOS image sensors with feature extraction

Patent metadata

Field	Value
Publication number	US-9936132-B2
Application number	US-201314064311-A
Country	US
Kind code	B2
Filing date	Oct 28, 2013
Priority date	Oct 26, 2012
Publication date	Apr 3, 2018
Grant date	Apr 3, 2018

Abstract

A CMOS imaging sensor with embedded feature extraction capability operatable in different modes by a method that includes the steps of: (a) operating the CMOS imaging sensor in a motion-detecting mode at a first power level using circuitry on the imaging sensor that generates motion data based on received images detected by pixels in the pixel array; (b) switching the imaging sensor from the motion-detecting mode to a feature extraction mode in response to detecting motion; and (c) operating the imaging sensor in the feature extraction mode at a second power level that is higher than the first power level.

First claim

The invention claimed is:

1. A method of operating a complementary metal oxide semiconductor (CMOS) imaging sensor having a pixel array and pixel processing circuitry, comprising the steps of:
(a) operating the CMOS imaging sensor in an in-pixel motion-detecting mode at a first power level using circuitry on the imaging sensor that generates motion data based on received images detected by pixels in the pixel array, wherein when in the in-pixel motion-detecting mode, a pair of pixels in the pixel array are merged and a previous frame signal generated by the merged pixels and a current frame signal generated by the merged pixels are compared by an in-pixel comparator to determine a frame difference therebetween, and further wherein a quantized frame-difference signal is generated by the in-pixel comparator when motion is detected based on the frame difference;
(b) switching the imaging sensor from the motion-detecting mode to a feature extraction mode in response to detecting motion; and
(c) operating the imaging sensor in the feature extraction mode at a second power level that is higher than the first power level, wherein operating in the feature extraction mode comprises:
receiving pixel data from the pixel array;
calculating spatial gradients based on the received pixel data;
determining angular information associated with the spatial gradients using a gradient-to-analog converter comprising mixed-signal processing circuitry by:
converting the spatial gradients into analog values;
generating an analog trigonometric value of the spatial gradients; and
determining angular information associated with the spatial gradients based on the analog trigonometric value; and

determining magnitudes of the spatial gradients associated with the angular information;

wherein the imaging sensor generates extracted feature data from one or more received images when operating in the feature extraction mode.

2. The method of claim 1 , further comprising the step of (d) switching from the feature extraction mode to an image capture mode based on analysis of the extracted feature data.

3. The method of claim 2 , further comprising the steps of (e) operating the imaging sensor in the image capture mode and storing or transmitting received images at a third power level that is higher than the second power level.

4. The method of claim 2 , wherein step (d) is carried out in response to detecting an object in one of the received images.

5. The method of claim 2 , wherein the extracted feature data comprises histograms of oriented gradients generated from the one or more received images.

6. The method of claim 1 , wherein step (a) further comprises generating 1 bit motion data for at least some of the pixels in the pixel array.

7. The method of claim 6 , wherein step (a) further comprises generating the 1 bit motion data for a subset of the pixels used for generating the extracted feature data.

8. An imaging device having a CMOS imaging sensor that operates according to the method of claim 1 .

9. The method of claim 1 , wherein receiving pixel data comprises receiving a plurality of rows or columns of pixel data from the pixel array; and wherein
calculating spatial gradients based on the received pixel data comprises
calculating spatial gradients based on the received rows or columns of pixel data.

10. The method of claim 9 , wherein determining magnitudes of the spatial gradients associated with the angular information comprises generating a histogram of gradients.

11. The method of claim 1 , wherein when in the motion-detecting mode, the method further comprises:
sampling, by a first in-pixel capacitor of one of the pair of pixels, the previous frame signal; and
sampling, by a second in-pixel capacitor of the other of the pair of pixels, the current frame signal, and
further wherein the in-pixel comparator determines the frame difference between the previous and current frame signals stored by the first and second in-pixel capacitors, respectively.

12. The method of claim 11 , wherein the previous frame signal is sampled by connecting the two pixels of the pair of pixels to the first in-pixel capacitor, and the current frame signal is sampled by connecting the two pixels of the pair of pixels to the second in-pixel capacitor.

13. A method of operating a complementary metal oxide semiconductor (CMOS) imaging sensor having a pixel array and pixel processing circuitry in an in-pixel motion detecting mode using circuitry on the imaging sensor that generates motion data based on received images detected by pixels in the pixel array, comprising:
sampling, by a first in-pixel capacitor of one of a pair of pixels, an analog previous frame signal, wherein the analog previous frame signal is sampled by connecting the two pixels of the pair of pixels to the first in-pixel capacitor;
sampling, by a second in-pixel capacitor of the other of the pair of pixels, an analog current frame signal, wherein the analog current frame signal is sampled by connecting the two pixels of the pair of pixels to the second in-pixel capacitor;
determining, by one or more in-pixel transistors reconfigured as an in-pixel comparator, a frame difference between the analog previous and current frame signals sampled by the first and second in-pixel capacitors, respectively; and
generating, by the one or more in-pixel transistors reconfigured as the in-pixel comparator, a quantized digital frame-difference signal when motion is detected based on the frame difference.

14. The method of claim 13 , wherein the generating step comprises generating a 1-bit frame-difference signal when motion is detected.

15. An imaging device having a CMOS imaging sensor that operates according to the method of claim 13 .

Assignees

• Univ Michigan Regents

Inventors

• Yoon Euisik
• Choi Jaehyuk

Classifications

• H04N23/667

  Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes · CPC title

• G06V10/955 Primary

  using specific electronic processors · CPC title

• H04N23/681

  Motion detection · CPC title

• H04N25/42

  by switching between different modes of operation using different resolutions or aspect ratios, e.g. switching between interlaced and non-interlaced mode · CPC title

• H04N25/771

  comprising storage means other than floating diffusion · CPC title