Tracking-based motion deblurring via coded exposure

Patent metadata

Field	Value
Publication number	US-11880984-B2
Application number	US-202117348392-A
Country	US
Kind code	B2
Filing date	Jun 15, 2021
Priority date	Jun 15, 2020
Publication date	Jan 23, 2024
Grant date	Jan 23, 2024

Abstract

Tracking-based motion deblurring via coded exposure is provided. Fast object tracking is useful for a variety of applications in surveillance, autonomous vehicles, and remote sensing. In particular, there is a need to have these algorithms embedded on specialized hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), to ensure energy-efficient operation while saving on latency, bandwidth, and memory access/storage. In an exemplary aspect, an object tracker is used to track motion of one or more objects in a scene captured by an image sensor. The object tracker is coupled with coded exposure of the image sensor, which modulates photodiodes in the image sensor with a known exposure function (e.g., based on the object tracking). This allows for motion blur to be encoded in a characteristic manner in image data captured by the image sensor. Then, in post-processing, deblurring is performed using a computational algorithm.

First claim

What is claimed is:

1. A method for tracking-based motion deblurring, the method comprising:
providing an image sensor in communication with a processor;
accessing, by the processor, image data associated with a scene captured by the image sensor;
tracking, by the processor, motion associated with a blurred object in the scene over multiple frames of the image data; and
performing coded exposure deblurring of the scene, including
modulating one or more photodiodes of the image sensor to track the blurred object in the scene across the multiple frames of the image data, and
encoding motion blur associated with the blurred object in the scene from the image data captured by the image sensor as the blurred object is tracked across the multiple frames of the image data,

wherein deblurring of the blurred object of the scene is accommodated by the motion blur as encoded and occurs subsequent to the tracking and identification of the blurred object, and
wherein tracking the blurred object comprises applying mean shift tracking to the image data including computing a color histogram for every frame of the multiple frames of the image data to find a centroid of the blurred object, the mean shift tracking accommodating automatic identification of the blurred object prior to deconvolution.

2. The method of claim 1 , further comprising identifying a region of interest (ROI) in the image data, the ROI comprising the blurred object.

3. The method of claim 2 , wherein the ROI is identified using an object detector.

4. The method of claim 2 , wherein tracking the blurred object further comprises tracking the ROI over multiple image frames in the image data.

5. The method of claim 1 , wherein tracking the blurred object in the scene comprises performing a tracking algorithm on image sensor hardware.

6. The method of claim 5 , further comprising performing coded exposure of an image sensor based on the tracking algorithm.

7. The method of claim 1 , wherein the method is performed on an embedded image processing system.

Assignees

• Jayasuriya Suren
• Iqbal Odrika
• Spanias Andreas
• Univ Arizona State

Inventors

• Jayasuriya Suren
• Iqbal Odrika
• Spanias Andreas

Classifications

• H04N19/20

  using video object coding · CPC title

• G06T7/20 Primary

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06T5/003

  Physics · mapped topic

• H04N23/73

  by influencing the exposure time · CPC title

• G06T2207/20208

  High dynamic range [HDR] image processing · CPC title