Computational imaging of the electric grid

Patent metadata

Field	Value
Publication number	US-11212459-B2
Application number	US-201816614432-A
Country	US
Kind code	B2
Filing date	May 16, 2018
Priority date	May 16, 2017
Publication date	Dec 28, 2021
Grant date	Dec 28, 2021

Abstract

Systems and methods for imaging scenes illuminated by light sources that are powered by alternating current. Data concerning these light sources are extracted from the imagery. Systems comprising a rolling shutter imaging sensor and configured to de-flicker images with spatial flicker are also provided.

First claim

The invention claimed is:

1. A system comprising:
an imaging device comprising a rolling shutter imaging sensor;
at least one hardware processor; and
a non-transitory computer-readable storage medium having stored thereon program instructions, the program instructions executable by the at least one hardware processor to:
operate the imaging device to capture at least one image of a scene illuminated at least by a light source powered by an AC (Alternating Current) power source,
generate a de-flickered version of said at least one image by extracting, for each pixel in said image, a flicker-independent intensity value,
estimate, in said at least one image, a set of pixel intensity values for a vertical or horizontal sequence of pixels corresponding to one flicker cycle,
compare said set of pixel intensity values to a database of sets of pixel intensity values corresponding to a plurality of light sources, and
identify said light source as one of said plurality of light sources in the database, wherein said estimating is based, at least in part, on applying to said image at least one digital bandstop filter having a frequency response which blocks multiple harmonics of a base frequency given as a multiple of (i) an inter-row or inter-column delay value of said rolling shutter, divided by (ii) a duration of a cycle of said AC power source.

2. The system of claim 1 , wherein said identifying comprises determining one or more characteristics of said AC power source selected from the group consisting of: a temporal point in said flicker cycle, a phase of said AC power source, phase changes of said AC power source, frequency fluctuations of said AC power source, amplitude of said AC power source, amplitude fluctuations of said AC power source, and nonlinearities in said AC power source.

3. The system of claim 1 , wherein said identifying comprises one of: (i) comparing said set of pixel intensity values to a database of sets of pixel intensity values corresponding to a plurality of light sources, and (ii) applying a machine learning classifier to a set of pixel intensity values for a vertical or horizontal sequence of pixels in said at least one image corresponding to one flicker cycle, wherein the classifier was trained on a plurality of sets of pixel intensity values corresponding to a plurality of labeled light sources.

4. The system of claim 1 , wherein said at least one image comprises a first image and a second image, wherein an exposure time used for said capturing of said second image is equal to an integer multiple of said flicker cycle, and wherein said estimating is based, at least in part, on said second image.

5. The system of claim 1 , wherein said light source is within direct view of said imaging device, and wherein said image is captured using an optical filter operatively coupled to said imaging device, wherein said optical filter has a transfer function that creates stripes in the captured image.

6. The system of claim 1 , (i) wherein said instructions further comprise operating said imaging device to capture a plurality of images, and (ii) wherein said estimating is based, at least in part, on estimating a temporal difference between each pair of images, based at least in part on relative intensity values of corresponding pixels in said pair of images.

7. The system of claim 1 , wherein said instructions further comprise operating said imaging device to capture a plurality of images over a specified period, and wherein said identifying further comprises generating a global-shutter image at any point during said period, based, at least in part, on estimating a Fourier series coefficients of a temporal signal observed at each pixel in said images.

8. The system of claim 1 , wherein said system is further configured for creating and storing a database of light sources, based, at least in part, on said detecting, wherein said database further comprises, for each light source, at least one of a type, a manufacturer, and a power rating.

9. A system comprising:
an imaging device comprising an imaging sensor and a shutter configured to move between a closed position and an open position;
at least one hardware processor; and
a non-transitory computer-readable storage medium having stored thereon program instructions, the program instructions executable by the at least one hardware processor to:
generate a signal having level changes corresponding to specified points in a cycle of a waveform detected from a mains alternate current (AC) outlet,
move said shutter into said open position based upon said level changes, for a specified number of said cycles,
operate the imaging device to capture image data of a scene illuminated by a plurality of light sources, wherein said image data is being captured when said shutter is in said open position during said specified number of cycles, and
extract, from said image data, components related only to a selected one of said plurality of light sources, wherein said extraction is based on detecting at least some of (i) a light response function, (ii) a chromaticity value, and (iii) an electric phase of said light source,

operatively coupled to said shutter, and wherein said capturing comprises defining a sequence of one or more of said shutter masks, wherein each of said shutter masks is applied to said shutter in said sequence, during each period when said shutter is in a said open position within a said cycle.

10. The system of claim 9 , wherein each of said open position has a duration of at least 1 microsecond, and wherein said specified number of cycles is at least 2.

11. A method comprising:
operating a rolling shutter imaging sensor to capture one or more images of a scene that is illuminated, at least in part, by a light source powered by an AC power source;
estimating pixel intensity values for a vertical or horizontal sequence of pixels in the one or more images, the sequence corresponding to a flicker cycle of the light source;
determining an identity of the light source, based on the estimation; and
performing at least one of:
(a) color-correcting at least one of the one or more images, based on the determined identity of the AC-powered light source;

(b) determining a phase of the AC power, based on the estimation; and
(c) when the scene is illuminated by more than one light source, at least one of which is AC-powered, creating at least one new images of the scene, said new image simulating an appearance of the scene under illumination by not all of the more than one light source.

12. The method of claim 11 , further comprising generating a de-flickered version of an image of said one or more image by extracting, for each pixel in said image, a flicker-independent intensity value, and wherein said estimating is based, at least in part, on applying to said image at least one digital bandstop filter having a frequency response which blocks multiple harmonics of a base frequency given as a multiple of (i) an inter-row or inter-column delay value of said rolling shutter, divided by (ii) a duration of a cycle of said AC power source.

13. The method of claim 11 , wherein said determining of an identity comprises determining one or more characteristics of said AC power source selected from the group consisting of: a temporal point in said flicker cycle, a phase of said AC power source, phase changes of said AC power source, frequency fluctuations of said AC power source, amplitude of said AC power source, amplitude fluctuations of said AC power source, and nonlinearities in said AC power source.

14. The method of claim 11 , wherein determining an identity further comprises

Assignees

• Technion Res & Dev Foundation
• Governing Council Univ Toronto

Inventors

• Sheinin Mark
• Schechner Yoav
• Kutulakos Kiriakos Neoklis
• Levron Yoash

Classifications

• G06V10/60 Primary

  relating to illumination properties, e.g. using a reflectance or lighting model · CPC title

• H04N23/745

  Detection of flicker frequency or suppression of flicker wherein the flicker is caused by illumination, e.g. due to fluorescent tube illumination or pulsed LED illumination · CPC title

• G06F2218/12

  Classification; Matching · CPC title

• H04N25/531

  by controlling rolling shutters in CMOS SSIS · CPC title

• H04N5/2357 Primary

  Electricity · mapped topic