Tracking-based motion deblurring via coded exposure

Patent metadata

Field	Value
Publication number	US-12437419-B2
Application number	US-202318516679-A
Country	US
Kind code	B2
Filing date	Nov 21, 2023
Priority date	Jun 15, 2020
Publication date	Oct 7, 2025
Grant date	Oct 7, 2025

Abstract

Tracking-based motion deblurring via coded exposure is provided. Fast object tracking is useful for a variety of applications in surveillance, autonomous vehicles, and remote sensing. In particular, there is a need to have these algorithms embedded on specialized hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), to ensure energy-efficient operation while saving on latency, bandwidth, and memory access/storage. In an exemplary aspect, an object tracker is used to track motion of one or more objects in a scene captured by an image sensor. The object tracker is coupled with coded exposure of the image sensor, which modulates photodiodes in the image sensor with a known exposure function (e.g., based on the object tracking). This allows for motion blur to be encoded in a characteristic manner in image data captured by the image sensor. Then, in post-processing, deblurring is performed using a computational algorithm.

First claim

What is claimed is:

1. An imaging device, comprising:
an image sensor; and
an image processor coupled to an optical sensor and configured to:
receive image data capturing a scene from the image sensor;
track a region of interest (ROI) corresponding to a blurred object across multiple frames in the image data;
perform coded exposure deblurring of the scene, to:
modulate one or more photodiodes of the image sensor based on the tracked ROI across the multiple frames of the image data; and
encode motion blur associated with the ROI in the scene from the image data captured by the image sensor as the ROI is tracked across the multiple frames of the image data,

wherein deblurring of the ROI in the scene is performed subsequent to the tracking and identification of the ROI, and
wherein mean shift tracking is employed to track the ROI to locate the ROI in each frame.

2. The imaging device of claim 1 , wherein the image processor is further configured to track the ROI by implementing a tracking algorithm.

3. The imaging device of claim 2 , wherein mean shift tracking includes estimation of how color histogram values of the ROI move as a cluster over time in the image data using non-parametric methods, wherein an object detector selects the ROI for the mean shift tracking, and, after receiving an initial bounding box, a color histogram is computed for each frame and the color histogram is used to find a centroid of the ROI.

4. The imaging device of claim 1 , wherein the image processor is further configured to deblur the scene based on the coded exposure of the image sensor.

5. The imaging device of claim 1 , wherein the image processor is an embedded image processor.

6. The imaging device of claim 5 , wherein the image processor comprises a field-programmable gate array (FPGA).

7. The imaging device of claim 5 , wherein the image processor comprises an application-specific integrated circuit (ASIC).

Assignees

• Jayasuriya Suren
• Iqbal Odrika
• Spanias Andreas
• Univ Arizona State

Inventors

• Jayasuriya Suren
• Iqbal Odrika
• Spanias Andreas

Classifications

• H04N19/20

  using video object coding · CPC title

• G06T5/73

  Deblurring; Sharpening · CPC title

• H04N23/73

  by influencing the exposure time · CPC title

• H04N23/75

  by influencing optical camera components · CPC title

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title