Pixel patch collection for prediction in video coding system

Patent metadata

Field	Value
Publication number	US-10536726-B2
Application number	US-201213463547-A
Country	US
Kind code	B2
Filing date	May 3, 2012
Priority date	Feb 24, 2012
Publication date	Jan 14, 2020
Grant date	Jan 14, 2020

Abstract

In a video coding system, an encoder may include a coding engine to predictively code input video, a decoder to reconstruct reference pictures generated by the coding engine, a reference picture cache to store the reconstructed reference pictures, a patch cache to store prediction patches generated from other sources, and a prediction search unit to search among the reference picture cache and the patch cache to generate prediction references for use by the coding engine while coding input video. The prediction patches may be assembled from a variety of sources including: predefined image content, reference pictures being evicted from the reference picture cache, image content of prior coding sessions and image data stored by applications on a common terminal where the encoder resides. A decoder may store prediction patches in its own patch cache for synchronous decoding.

First claim

We claim:

1. An encoder apparatus, comprising:
a reference picture cache that stores decoded reference pictures;
a patch cache that stores prediction patches; and
a processor executing instructions stored in a memory that cause:
exchanging signaling with a remote decoder to establish a coding session,
prior to exchanging coded video data with the remote decoder pursuant to the coding session, storing video content in the patch cache of the encoder apparatus,
after the coding session is established:
coding an input video sequence according to predictive coding techniques;
decoding the coded video data of reference pictures generated by the coding and storing the decoded reference pictures in the reference picture cache of the encoder apparatus;
as part of the coding of a video element of the input video sequence, generating a prediction reference for the video element:
first searching, from among the reference picture cache, for the prediction reference for the video element,
if the prediction reference is not found in the reference picture cache, second searching, from among the patch cache for the prediction reference for the video element, and selecting, for the video element, the prediction reference from results of the first and second searchings, wherein the coding of the video element uses the selected prediction reference as a basis of the predictive coding techniques; and
transmitting the coded video data to a channel.

2. The encoder of claim 1 , wherein the stored prediction patches are derived from an identification of a predetermined type of an object.

3. The encoder of claim 2 , wherein the object type is a face.

4. The encoder of claim 2 , wherein the object type is a text.

5. The encoder of claim 1 , wherein the stored prediction patches are derived from application data stored within a terminal.

6. The encoder of claim 1 , wherein the stored prediction patches are predefined at a terminal prior to an onset of a video coding session.

7. The encoder of claim 1 , wherein the stored prediction patches are derived from a prior video coding session.

8. The encoder of claim 1 , wherein a prediction reference selector searches among the reference picture cache first to identify the prediction references for the input video to be coded and searches among the patch cache second.

9. The encoder of claim 1 , wherein the prediction reference selector searches among the reference picture cache and the patch cache in an order determined by an object detector.

10. The encoder of claim 1 , wherein the prediction selector searches the patch cache if a low correlation is achieved by searching the reference picture cache.

11. The encoder of claim 1 , wherein data is stored in the patch cache for a longer duration than data in the reference picture cache.

12. The encoder of claim 1 , wherein the stored prediction patches are derived from a data of a contacts application installed on a device that includes the encoder apparatus.

13. The encoder of claim 1 , wherein the stored prediction patches are derived from a data of a photo management application installed on a device that includes the encoder apparatus.

14. The encoder of claim 1 , wherein a source of prediction patches include decoded frames from a previously-terminated coding session.

15. The encoder of claim 1 , wherein the prediction patches include video content.

16. A decoder apparatus, comprising:
a reference picture cache that stores decoded reference pictures;
a patch cache that stores prediction patches; and
a processor executing instructions stored in a memory that cause:
exchanging signaling with an encoder to establish a decoding session,
prior to exchanging coded video data with the encoder pursuant to the decoding session, storing video content in the patch cache of the decoder apparatus,
decoding a coded video sequence, received from the encoder, according to predictive coding techniques,
storing decoded data for the decoded reference pictures in the reference picture cache, and
as part of the decoding of a video element of the coded video data, generating a prediction for the video element by retrieving data from one of the reference picture cache and the patch cache based on a prediction reference included in the coded video sequence, wherein the decoding of the video element uses the selected prediction reference as a basis of the predictive decoding.

17. The decoder of claim 16 , wherein the stored prediction patches are derived from a predetermined type of an object.

18. The decoder of claim 17 , wherein the object type is a face.

19. The decoder of claim 17 , wherein the object type is a text.

20. The decoder of claim 16 , wherein the stored prediction patches are derived from application data stored within a terminal.

21. The decoder of claim 16 , wherein the stored prediction patches are derived from the coded video data.

22. The decoder of claim 16 , wherein the stored prediction patches are predefined at a terminal prior to an onset of a decoding session.

23. The decoder of claim 16 , wherein the stored prediction patches are derived from a prior video coding session.

24. The decoder of claim 16 , wherein the stored prediction patches are derived from a data of a contacts application installed on a device that includes the decoder apparatus.

25. The decoder of claim 16 , wherein the stored prediction patches are derived from a data of a photo management application installed on a device that includes the decoder apparatus.

26. The decoder of claim 16 , wherein a source of prediction patches include decoded frames from a previously-terminated decoding session.

27. A non-transitory computer readable medium storing program instructions that, when executed by a processor, cause the processor to perform a video coding method, comprising:
exchanging signaling with a remote decoder to establish a coding session,
prior to exchanging coded video data with the remote decoder pursuant to the coding session, storing video content in the patch cache of an encoder,
after the coding session is established:
coding an input video sequence according to predictive coding techniques;
decoding the coded video data of reference pictures generated by the coding and storing the decoded reference pictures in the reference picture cache of the encoder;
as part of the coding of a video element of the input video sequence, generating a prediction reference for the video element:
first searching, from among the reference picture cache, for the prediction reference for the video element,
if the prediction reference is not found in the reference picture cache, second searching, from among the patch cache for the prediction reference for the video element, and
selecting, for the video element, the prediction reference from results of the first and second searchings, wherein the coding of the video element uses the selected prediction reference as a basis of the predictive coding techniques; and
transmitting the coded video data to a channel.

28. The medium of claim 27 , wherein the stored video content is derived from a data of a contacts application installed on a device.

29. The medium of claim 27 , wherein the stored video content is derived from a data of a photo management application installed on a device.

30. The medium of clai

Assignees

• Zhou Xiaosong
• Normile James Oliver
• Wu Hsi Jung
• Apple Inc

Inventors

• Zhou Xiaosong
• Normile James Oliver
• Wu Hsi-Jung

Classifications

• H04N19/103

  Selection of coding mode or of prediction mode · CPC title

• H04N19/105

  Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction · CPC title

• H04N19/176

  the region being a block, e.g. a macroblock · CPC title

• H04N19/20

  using video object coding · CPC title

• H04N19/97 Primary

  Matching pursuit coding · CPC title