Tuning video compression for high frame rate and variable frame rate capture

Patent metadata

Field	Value
Publication number	US-10009628-B2
Application number	US-201313972434-A
Country	US
Kind code	B2
Filing date	Aug 21, 2013
Priority date	Jun 7, 2013
Publication date	Jun 26, 2018
Grant date	Jun 26, 2018

Abstract

Video data with high frame rates may be displayed on devices with limited resources (e.g., decoder and/or display resources). These devices may have their resources devoted to other tasks or may not be capable to display the video data at the high frame rates. The coding method may include coding the frames such that additional droppable frames are included in the encoded video data. The decoding method may include dropping droppable frames before the encoded video data is decoded to reduce the number of frames that will be decoded and displayed. These methods may be applied to video data that has a variable frame rate and may be combined with processing the image sequence for slow motion playback.

First claim

We claim:

1. A coding method, comprising:
coding, by the encoder, an image sequence according to predictive coding techniques in which select coded frames serve as prediction references for other coded frames, wherein the coding comprises:
distinguishing first frames from the image sequence that are to be displayed according to a display rate of a normal playback mode at a decoder from additional droppable frames of the same image sequence associated with a slow motion playback mode that are to be displayed at the decoder at a display rate that is higher than the display rate of the normal playback mode,
for the first frames, coding the first frames according to the predictive coding techniques in which the respective first frames are candidates to serve as prediction references for other first frames of the image sequence, and
for the additional droppable frames, coding the additional droppable frames according to the predictive coding techniques using reference frames from the first frames as prediction references, wherein the additional droppable frames are prevented from serving as prediction references for any of the first frames of the image sequence, such that at least one additional droppable frame can be retimed and presented in slow motion at a decoder.

2. The method of claim 1 , wherein the image sequence includes a plurality of frames having a frame rate that exceeds the display rate of the normal playback mode.

3. The method of claim 1 , wherein the image sequence includes a plurality of frames having a frame rate that at least matches the display rate of the enhanced playback mode.

4. The method of claim 1 further comprising transmitting the coded image sequence over a communication channel.

5. The method of claim 1 , wherein the image sequence has a variable frame rate.

6. The method of claim 1 , wherein the display of the normal playback mode rate is a frame rate used to display the image sequence on a display device.

7. The method of claim 1 , wherein the remaining frames include an intra-frame (I-Frame), a predictive frame (P-Frame) and a bidirectional frame (B-Frame).

8. The method of claim 1 , further comprising:
transmitting the coded image sequence; and
decoding the coded image sequence.

9. A decoding method, comprising:
decoding a coded image sequence according to predictive decoding techniques in which select coded frames are coded using other coded frames as prediction references, wherein the decoding comprises:
distinguishing first frames from the image sequence that are to be displayed according to a display rate of a normal playback mode at a decoder from additional droppable frames of the same image sequence associated with a slow motion playback mode that are to be displayed at the decoder at a display rate that is higher than the display rate of the normal playback mode,
for the first frames, decoding the first frames according to the predictive coding techniques in which the respective first frames are candidates to serve as prediction references for other first frames of the image sequence,
for the additional droppable frames, decoding the additional droppable frames according to the predictive coding techniques in which the remaining frames are decoded predictively using reference frames from the first frames as prediction references, wherein the additional droppable frames are prevented from serving as prediction references for any of the first frames of the image sequence,
for a slow motion portion of the coded image sequence, retiming frames according to a slow motion playback rate,
presenting decoded first frames from outside the slow motion portion at the display rate of normal playback mode, and
presenting retimed first frames and at least one retimed additional droppable frame from the slow motion portion at the display rate of normal playback mode.

10. The method of claim 9 , wherein the coded image sequence includes a plurality of frames having a frame rate that exceeds the display rate of the normal playback mode.

11. The method of claim 9 , wherein the coded image sequence has a variable frame rate.

12. The method of claim 9 , wherein the remaining frames include an intra-frame (I-Frame), a predictive frame (P-Frame) and a bidirectional frame (B-Frame).

13. The method of claim 9 , wherein the display rate of the enhanced playback mode is selected based on resources available for decoding the coded image sequence.

14. The method of claim 9 , wherein the display rate of the enhanced playback mode is selected for a portion of the coded image sequence to be displayed with slow motion.

15. The method of claim 9 , further comprising:
transmitting the coded image sequence; and
decoding the coded image sequence.

16. A non-transitory storage device that stores a predictively-coded image sequence comprising a plurality of coded frames, wherein select coded frames serve as prediction references for other coded frames, first frames are distinguished from the image sequence that are to be displayed according to a display rate of a normal playback mode from additional droppable frames of the image sequence associated with a slow motion playback mode that are to be displayed at a display rate that is higher than the display rate of the normal playback mode, the first frames are coded according to the predictive coding techniques in which the respective first frames are candidates to serve as prediction references for other first frames of the same image sequence, and the additional droppable frames are coded according to the predictive coding techniques using reference frames from the first frames as prediction references, wherein the additional droppable frames are prevented from serving as prediction references for any of the first frames of the same image sequence such that at least one additional droppable frame can be retimed and presented in slow motion at a decoder.

17. The device of claim 16 , wherein the coded frames are transmitted and decoded.

18. A method for processing a coded image sequence for slow motion playback at a display, the method comprising:
receiving a coded image sequence with a frame rate of an enhanced playback mode that is higher than a display rate of a normal playback mode of a display that will display a decoded image sequence;
receiving a command selecting a portion of the coded image sequence for slow motion playback at the display;
decoding the frames in the selected portion according to predictive coding techniques in which the frames in the selected portion are candidates to serve as prediction references for other first frames of the same image sequence; and
retiming durations of the decoded frames within the selected portion for slow motion playback from a duration of the higher frame rate of the enhanced playback mode to a display duration of frames at the display rate of the normal playback mode;
decoding remaining frames outside the selected portion according to the predictive coding techniques using reference frames from the first frames as prediction references, wherein certain of the remaining frames outside the selected portion that do not serve as prediction references for first frames of the image sequence are dropped prior to displaying at the display;
presenting decoded first frames from outside the portion for slow motion at the display rate of the normal playback mode; and
presenting retimed first frames and at least one retimed additional droppable frame from the portion for slow motion at the display rate of normal playback mode.

19.

Assignees

• Apple Inc

Inventors

• Bushell John Samuel
• Concion Davide
• Saracino David
• Turnquist Erik

Classifications

• H04N19/172

  the region being a picture, frame or field · CPC title

• H04N19/107

  between spatial and temporal predictive coding, e.g. picture refresh · CPC title

• H04N19/16

  for a given display mode, e.g. for interlaced or progressive display mode · CPC title

• H04N19/577

  Motion compensation with bidirectional frame interpolation, i.e. using B-pictures · CPC title

• H04N19/132

  Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking · CPC title