Systems and methods for post processing time-lapse videos

Patent metadata

Field	Value
Publication number	US-10133934-B2
Application number	US-201615273644-A
Country	US
Kind code	B2
Filing date	Sep 22, 2016
Priority date	Sep 30, 2015
Publication date	Nov 20, 2018
Grant date	Nov 20, 2018

Abstract

Techniques and devices for post-processing time-lapse videos are described. The techniques include obtaining an input time-lapse sequence of frames and determining a visual metric value, e.g., average luminance, for each frame. A curve of best fit may then be determined for the visual metric values of the frames. The visual metric values, e.g., the average luminance values, of the plurality of frames may then be adjusted, e.g., by adjusting the visual metric values of each frame to be equal to the corresponding value determined by the curve of best fit. Some embodiments include further adjusting the visual metric values to be equal to a weighted average of the adjusted visual metric values for adjacent frames in the time-lapse sequence. Finally, a visual characteristic of the frames, e.g., an image histogram, may be adjusted based on the frame's determined adjusted visual metric value, and an output time-lapse sequence may be generated.

First claim

The invention claimed is:

1. A non-transitory program storage device, readable by a programmable control device, comprising instructions stored thereon that, when executed, cause the programmable control device to:
obtain an input time-lapse sequence of frames;
determine a visual metric value for each frame of the input time-lapse sequence of frames;
determine a first curve to fit the visual metric values for the input time-lapse sequence of frames;
determine an adjusted visual metric value for one or more of the frames of the input time-lapse sequence of frames, wherein the determined adjusted visual metric value for each frame is equal to a corresponding value from the determined first curve, and wherein the one or more adjusted visual metric values comprise first adjusted visual metric values;
adjust a visual characteristic of the one or more of the frames of the input time-lapse sequence of frames, such that, after the adjustment of the visual characteristic of each of the one or more frames, the visual metric value of each of the one or more frames is moved towards its respective first adjusted visual metric value; and
generate an output time-lapse sequence of frames comprising the one or more frames having the adjusted visual characteristic.

2. The non-transitory program storage device of claim 1 , wherein the instructions that, when executed, cause the programmable control device to obtain the input time-lapse sequence of frames comprise instructions that, when executed, cause the programmable control device to:
obtain a first input time-lapse sequence of frames in a first format; and
convert the first input time-lapse sequence of frames into a second format to generate the input time-lapse sequence of frames.

3. The non-transitory program storage device of claim 2 , wherein:
the first format comprises an RGB format; and
the second format comprises aft YC B C R format.

4. The non-transitory program storage device of claim 1 , wherein the instructions that, when executed, cause the programmable control device to determine the visual metric value for each frame comprise instructions that, when executed, cause the programmable control device to determine an average luminance value for each frame of the input time-lapse sequence of frames.

5. The non-transitory program storage device of claim 1 , wherein the instructions that, when executed, cause the programmable control device to adjust a visual characteristic of the one or more of the frames of the input time-lapse sequence of frames further comprise instructions that, when executed, cause the programmable control device to:
adjust a visual characteristic of the one or more of the frames of the input time-lapse sequence of frames, such that, after the adjustment of the visual characteristic of each of the one or more frames, the visual metric value of each of the one or more frames is equal to its respective first adjusted visual metric value.

6. The non-transitory program storage device of claim 1 , wherein the instructions that, when executed, cause the programmable control device to adjust the visual metric value for one or more of the frames of the input time-lapse sequence of frames further comprise instructions that, when executed, cause the programmable control device to:
further adjust the first adjusted visual metric value of at least one of the one or more frames to be equal to a weighted average of the first adjusted visual metric values of one or more adjacent frames in the input time-lapse sequence of frames.

7. The non-transitory program storage device of claim 1 , wherein each frame of the input time-lapse sequence of frames comprises an image histogram, and
wherein the instructions that, when executed, cause the programmable control device to adjust a visual characteristic of the one or more of the frames of the input time-lapse sequence of frames further comprise instructions that, when executed, cause the programmable control device to shift the image histograms of the one or more frames based on the first adjusted visual metric value of the respective frame.

8. The non-transitory program storage device of claim 7 , wherein the instructions that, when executed, cause the programmable control device to adjust a visual characteristic of the one or more of the frames of the input time-lapse sequence of frames further comprise instructions that, when executed, cause the programmable control device to spread the image histograms of the one or more frames.

9. A computer-implemented method for generating a time-lapse sequence of frames, comprising:
obtaining an input time-lapse sequence of frames;
determining a visual metric value for each frame of the input time-lapse sequence of frames;
determining a first curve to fit the visual metric values for the input time-lapse sequence of frames;
determining an adjusted visual metric value for one or more of the frames of the input time-lapse sequence of frames, wherein the determined adjusted visual metric value for each frame is equal to a corresponding value from the determined first curve, and wherein the one or more adjusted visual metric values comprise first adjusted visual metric values;
adjusting a visual characteristic of the one or more of the frames of the input time-lapse sequence of frames, such that, after the adjustment of the visual characteristic of each of the one or more frames, the visual metric value of each of the one or more frames is moved towards its respective first adjusted visual metric value; and
generating an output time-lapse sequence of frames comprising the one or more frames having the adjusted visual characteristic.

10. The computer-implemented method of claim 9 , wherein obtaining the input time-lapse sequence of frames comprises:
obtaining a first input time-lapse sequence of frames in a first format; and
converting the first input time-lapse sequence of frames into a second format to generate the input time-lapse sequence of frame.

11. The computer-implemented method of claim 10 , wherein:
the first format comprises an RGB format; and
the second format comprises aft YC B C R format.

12. The computer-implemented method of claim 9 , wherein determining a visual metric value comprises determining an average luminance value of each frame in the input time-lapse sequence of frames.

13. The computer-implemented method of claim 9 , wherein adjusting a visual characteristic of the one or more of the frames of the input time-lapse sequence of frames further comprises:
adjusting a visual characteristic of the one or more of the frames of the input time-lapse sequence of frames, such that, after the adjustment of the visual characteristic of each of the one or more frames, the visual metric value of each of the one or more frames is equal to its respective first adjusted visual metric value.

14. The computer-implemented method of claim 9 , wherein adjusting the visual metric value for one or more of the frames of the input time-lapse sequence of frames further comprises:
further adjusting the first adjusted visual metric value of at least one of the one or more frames to be equal to a weighted average of the first adjusted visual metric values of one or more adjacent frames in the input time-lapse sequence of frames.

15. The computer-implemented method of claim 9 , wherein each frame of the input time-lapse sequence of frames comprises an image histogram, and
wherein adjusting a visual characteristic of the one or more of the frames of the input time-lapse sequence of frames further comprises shifting the image histograms of the one or more frames

Assignees

• Apple Inc

Inventors

• Wong Earl Q
• Johnson Garrett

Classifications

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G06K9/00744 Primary

  Physics · mapped topic

• H04N19/70

  characterised by syntax aspects related to video coding, e.g. related to compression standards · CPC title

• G06K9/00

  Physics · mapped topic

• H04N5/783

  Adaptations for reproducing at a rate different from the recording rate · CPC title