HDR/WDR image time stamps for sensor fusion

Patent metadata

Field	Value
Publication number	US-10218913-B2
Application number	US-201715490621-A
Country	US
Kind code	B2
Filing date	Apr 18, 2017
Priority date	Apr 18, 2017
Publication date	Feb 26, 2019
Grant date	Feb 26, 2019

Abstract

Provided are systems and methods for determining timestamp information for High Dynamic Range and/or Wide Dynamic Range composite images. For example, an apparatus is provided that comprises an image sensor configured to capture a plurality of sub-frames of a scene, wherein each sub-frame comprises an image of the scene captured using an exposure time that is different from at least one other exposure time of at least one other sub-frame of the plurality of sub-frames. The apparatus is configured to receive, for each of the plurality of sub-frames, sub-pixel image data corresponding to a first portion of an image frame, determine composite image data corresponding to the first portion of the image frame based on values of the received sub-pixel image data for the plurality of sub-frames, identify an indicator based on the sub-frames corresponding to the received sub-pixel image data, and determine timestamp information, based on the identified indicator.

First claim

What is claimed is:

1. An apparatus to determine timestamp information, the apparatus comprising:
an image sensor configured to capture a plurality of sub-frames of a scene, wherein each sub-frame comprises an image of the scene captured using an exposure time that is different from at least one other exposure time of at least one other sub-frame of the plurality of sub-frames; and
at least one processor coupled to the image sensor and configured to:
receive, from the image sensor, for each of the plurality of sub-frames, sub-pixel image data corresponding to a first portion of an image frame;
determine composite image data corresponding to the first portion of the image frame based on values of the received sub-pixel image data for the plurality of sub-frames and by selecting, from the plurality of sub-frames a particular sub-frame to be associated with the composite image data based on luminosity values of the sub-pixel image data of the plurality of sub-frames;
identify an indicator based on the sub-frames corresponding to the received sub-pixel image data used to determine the composite image data; and
determine timestamp information, based on the identified indicator, wherein the timestamp information corresponds to the composite image data based upon timing information for the plurality of sub-frames.

2. The apparatus of claim 1 , wherein the plurality of sub-frames are captured using a rolling shutter.

3. The apparatus of claim 1 , wherein the composite image data corresponds to high dynamic range (HDR) image data or wide dynamic range (WDR) image data.

4. The apparatus of claim 1 , wherein indicator indicates the particular sub-frame associated with the composite image data.

5. The apparatus of claim 1 , wherein to determine the timestamp information based on the identified indicator, the processor is configured to:
identify the timing information for the plurality of sub-frames, wherein the timing information comprises at least initial timing information associated with the plurality of sub-frames, and a rolling shutter angle associated with the image sensor;
determine sub-pixel timing information for each of the plurality of sub-frames, based upon the initial timing information associated with the plurality of sub-frames, a location of the first portion of the image frame, and the rolling shutter angle associated with the image sensor; and
determine a timestamp associated with the composite image data, based upon the determined sub-pixel timing information and the indicator associated with the composite image data.

6. The apparatus of claim 5 , wherein the sub-pixel timing information for each of the plurality of sub-frames is pre-computed and stored in a data structure.

7. The apparatus of claim 5 , wherein the location of the first portion of the image frame comprises a row of the image frame.

8. An apparatus to determine timestamp information, the apparatus comprising:
an image sensor configured to capture a plurality of sub-frames of a scene, wherein each sub-frame comprises an image of the scene captured using an exposure time that is different from at least one other exposure time of at least one other sub-frame of the plurality of sub-frames; and
at least one processor coupled to the image sensor and configure to:
receive, from the image sensor, for each of the plurality of sub-frames, sub-pixel image data corresponding to a first portion of an image frame;
determine composite image data corresponding to the first portion of the image frame based on values of the received sub-pixel image data for the plurality of sub-frames;
identify an indicator based on the sub-frames corresponding to the received sub-pixel image data used to determine the composite image data; and
determine timestamp information, based on the identified indicator, wherein the timestamp information corresponds to the composite image data based upon timing information for the plurality of sub-frames,

wherein determining the composite image data comprises blending values of the sub-pixel image data of one or more sub-frames of the plurality of sub-frames using one or more blending parameters, each blending parameter corresponding to a sub-frame of the plurality of sub-frames, and indicating a degree to which the sub-pixel image data corresponding to the sub-frame is used to determine the composite image data, and wherein the indicator indicates at least a portion of the one or more blending parameters.

9. The apparatus of claim 8 , wherein a sum of the one or more blending parameters is equal to 1.

10. A method for determining timestamp information for wide dynamic range composite images, comprising:
receiving, from an image sensor, sub-pixel image data corresponding to a first portion of an image frame for each of a plurality of sub-frames, each sub-frame comprising an image of a scene captured using an exposure time that is different from at least one other exposure time of at least one other sub-frame of the plurality of sub-frames;
determining composite image data corresponding to the first portion of the image frame based on values of the received sub-pixel image data for the plurality of sub-frames and by selecting, from the plurality of sub-frames, a particular sub-frame to be associated with the composite image data based on luminosity values of the sub-pixel image data of the plurality of sub-frames;
identifying an indicator based on the sub-frames corresponding to the received sub-pixel image data used to determine the composite image data; and
determining timestamp information, based on the identified indicator, wherein the timestamp information corresponds to the composite image data based upon timing information for the plurality of sub-frames.

11. The method of claim 10 , wherein the sub-pixel image data received from the image sensor is captured using a rolling shutter.

12. The method of claim 10 , wherein the composite image data corresponds to high dynamic range (HDR) image data or wide dynamic range (WDR) image data.

13. The method of claim 10 , wherein the composite image data corresponds to wide dynamic range (WDR) image data, and wherein the indicator indicates the particular sub-frame associated with the WDR image data.

14. The method of claim 10 , wherein determining composite image data comprises blending sub-pixel image data of one or more sub-frames of the plurality of sub-frames using one or more blending parameters, each blending parameter corresponding to a sub-frame of the plurality of sub-frames, wherein the composite image data corresponds to wide dynamic range (WDR) image data, and indicating a degree to which the sub-pixel image data corresponding to the sub-frame is used to determine the WDR image data, and wherein the indicator indicates at least a portion of the one or more blending parameters.

15. The method of claim 14 , wherein a sum of the one or more blending parameters is equal to 1.

16. A method for determining timestamp information for wide dynamic range composite images, comprising:
receiving, from an image sensor, sub-pixel image data corresponding to a first portion of an image frame for each of a plurality of sub-frames, each sub-frame comprising an image of a scene captured using an exposure time that is different from at least one other exposure time of at least one other sub-frame of the plurality of sub-frames;
determining composite image data corresponding to the first portion of the image frame based on values of the received sub-pixel image data for the plurality of sub-frame;
identifying an indicator based on the sub-frames corresponding t

Assignees

• Qualcomm Inc

Inventors

• Somasundaram Kiran
• Biswas Mainak

Classifications

• H04N23/743

  Bracketing, i.e. taking a series of images with varying exposure conditions · CPC title

• H04N25/531

  by controlling rolling shutters in CMOS SSIS · CPC title

• H04N25/583

  with different integration times · CPC title

• H04N25/589

  with different integration times, e.g. short and long exposures · CPC title

• H04N23/741 Primary

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title