Method And Apparatus For Finding And Using Video Portions That Are Relevant To Adjacent Still Images

Patent metadata

Field	Value
Publication number	US-2016358634-A1
Application number	US-201514865752-A
Country	US
Kind code	A1
Filing date	Sep 25, 2015
Priority date	Jun 7, 2015
Publication date	Dec 8, 2016
Grant date	—

Abstract

The invention relates to systems, methods, and computer readable media for responding to a user snapshot request by capturing anticipatory pre-snapshot image data as well as post-snapshot image data. The captured information may be used, depending upon the embodiment, to create archival image information and image presentation information that is both useful and pleasing to a user. The captured information may automatically be trimmed or edited to facilitate creating an enhanced image, such as a moving still image. Varying embodiments of the invention offer techniques for trimming and editing based upon the following: exposure, brightness, focus, white balance, detected motion of the camera, substantive image analysis, detected sound, image metadata, and/or any combination of the foregoing.

First claim

We claim:

1 ) A method of producing an enhanced image comprising:
capturing an archive of image data comprising a plurality of image frames segmented as a pre-capture image sequence, a snapshot, and a post-capture image sequence, wherein the archival image data is captured by:
capturing the pre-capture image sequence,
receiving a snapshot indication through a user interface, the snapshot indication received temporally after capturing at least a portion of the pre-capture image sequence,
in response to the snapshot indication, capturing the snapshot image,
capturing the post-capture image sequence, wherein at least a portion of the post-capture image sequence is captured temporally after receiving the snapshot indication, and
capturing contextual information associated with the pre-capture image sequence, the snapshot, and the post-capture image sequence,
trimming or flagging the image frames of the pre-capture image sequence and the post-capture image sequence based upon an analysis of the contextual information and combined image frames of the archival image data, wherein the analysis comprises:
developing a target brightness level and an average brightness level,
for each image frame, computing a ratio of the target brightness level with the average brightness level,
trimming or flagging each pre-capture image frame and each post-capture image frame based upon a comparison of a ratio threshold to the image frame's ratio of target brightness level to average brightness level,
determining a velocity or acceleration associated with each image frame,
setting a threshold for velocity or acceleration based upon the velocity or acceleration of the snapshot image frame, and
trimming or flagging each pre-capture image frame based upon a comparison of its velocity or acceleration to the threshold for velocity or acceleration, and
creating an enhanced image based upon the archival image data and the analysis.

2 ) The method of claim 1 , wherein the target brightness level, and the average brightness level are metadata received from an auto exposure system.

3 ) The method of claim 1 , wherein the ratio threshold is set independently for each image frame.

4 ) The method of claim 1 , wherein the pre-capture image data begins to be captured upon the activation of a camera application.

5 ) The method of claim 1 , wherein the contextual information includes one or more of the following:
sensor output data; application status data; data indicating the presence of a face in an image frame; and camera metadata.

6 ) The method of claim 1 , wherein the analysis includes, for each image frame, determining whether a loud noise is associated with the image frame and flagging or trimming the image frame based upon the determination, and wherein flagging the image frame includes scoring the image frame.

7 ) The method of claim 1 , wherein the analysis comprises:
determining a multi-factor indicator that includes a camera travel distance indicator derived from image frame registration techniques, a blurriness indicator, and an image quality assessment indicator derived based upon image frame registration techniques; normalizing each of the camera travel distance indicator, the blurriness indicator, and the image quality assessment indicator; combining the normalized indicators to compute a value for the multi-factor indicator; and flagging or trimming each image frame based upon the value of the multi-factor indicator.

8 ) The method of claim 7 , wherein the archival image data comprises a video file and an associated still image file, and wherein the method further comprises displaying the still image, and upon receipt of a user indication, finding the associated video file and playing the associated video file.

9 ) A computer readable medium comprising one or more instructions that, when executed, configure a processor to:
capture an archive of image data comprising a plurality of image frames segmented as a pre-capture image sequence, a snapshot, and a post-capture image sequence, wherein the instructions that configure the processor to capture the archival image data comprise instructions that configure the processor to:
capture the pre-capture image sequence,
receive a snapshot indication through a user interface, the snapshot indication received temporally after capturing at least a portion of the pre-capture image sequence,
in response to the snapshot indication, capture the snapshot image,
capture the post-capture image sequence, wherein at least a portion of the post capture image sequence is captured temporally after receiving the snapshot indication, and
capture contextual information associated with the pre-capture image sequence, the snapshot, and the post-capture image sequence,
automatically trim or flag the image frames of the pre-capture image sequence and the post-capture image sequence based at least in part upon the contextual information; and create an enhanced image based upon the trimmed archival image data.

10 ) The computer readable medium of claim 9 wherein, the enhanced image is stored as a video file and an associated still image file.

11 ) The computer readable medium of claim 10 , further comprising instructions that configure the processor to play the enhanced image by displaying the snapshot, finding the associated video file, and playing the associated video file.

12 ) The computer readable medium of claim 9 , wherein the pre-capture image data begins to be captured upon the activation of a camera application.

13 ) The computer readable medium of claim 9 , wherein the contextual information includes one or more of the following:
sensor output data; application status data; data indicating the presence of a face in an image frame; camera metadata; data indicating the presence of animals; scene classification information; and information accessed over the Internet.

14 ) The computer readable medium of claim 9 , further comprising instructions that configure the processor to, for a plurality of image frames, determine whether a loud noise is associated with the image frame and flag or trim the image frame based upon the determination.

15 ) The computer readable medium of claim 9 , further comprising instructions that configure the processor to:
determine a multi-factor indicator that includes a camera travel distance indicator derived from image frame registration techniques, a blurriness indicator, and an image quality assessment indicator derived based upon image frame registration techniques; normalize each of the camera travel distance indicator, the blurriness indicator, and the image quality assessment indicator; combine the normalized indicators to compute a value for the multi-factor indicator; and flag or trim a plurality of image frames based upon the value of the multi-factor indicator.

16 ) A system comprising:
one or more CPUs; one or more cameras for capturing an archive of image data comprising a plurality of image frames segmented as a pre-capture image sequence, a snapshot, and a post-capture image sequence; a memory for storing program instructions for the one or more CPUs, where the instructions, when executed, cause the one or more CPUs to:
capture the pre-capture image sequence,
receive a snapshot indication, the snapshot indication received temporally after capturing at least a portion of the pre-capture image sequence,
in response to the snapshot indication, capture the snapshot image,
capture the post-capture image sequence, wherein at least a portion of the post cap

Assignees

• Apple Inc

Inventors

• Molgaard Claus
• Keating Brett M
• Williams George E
• Zuliani Marco
• Wong Vincent Y
• Doepke Frank
• Tira-Thompson Ethan J

Classifications

• H04N5/2621 Primary

  Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability · CPC title

• G11B27/34 Primary

  Indicating arrangements  {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)} · CPC title

• H04N23/741

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title

• H04N23/72

  Combination of two or more compensation controls · CPC title

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title