Method and system of frame re-ordering for video coding

Patent metadata

Field	Value
Publication number	US-11051026-B2
Application number	US-201514841624-A
Country	US
Kind code	B2
Filing date	Aug 31, 2015
Priority date	Aug 31, 2015
Publication date	Jun 29, 2021
Grant date	Jun 29, 2021

Abstract

Techniques related to frame re-ordering for video coding are described herein.

First claim

What is claimed is:

1. A computer-implemented method of video coding comprising:
receiving video frames of image data of a first video sequence having frames in a first order, wherein each video frame is a separate image;
performing a processing operation that modifies image data of the video frames to form local video frames;
directly after the processing operation, placing the local video frames of the first video sequence into an on-chip cache memory, said on-chip cache memory with a capacity of a region of a single frame at a time, wherein said region is less than an entire frame, and without first obtaining the local video frames from an off-chip memory;
receiving other frames of the first video sequence from the off-chip memory;
re-ordering the frames into a second video sequence having a second order different from the first order and comprising obtaining the local video frames via the on-chip memory without use of the off-chip memory and placing the local frames in the second video sequence according to the second order and with frames from the off-chip memory;
providing the frames in the second video sequence to code or display the image data;
wherein the first video sequence order is associated with a first rate and the second video sequence order is associated with a second rate different from the first rate; and
wherein individual display frame periods of the first video sequence order are replaced with multiple frames of the second video sequence order comprising placing one frame from on-chip memory and a copy of the same frame from off-chip memory respectively in two consecutive frame spaces of the second video sequence order.

2. The method of claim 1 wherein the first order is a captured order as the frames were captured by an image capturing device and the second order is a coding order of frames as the frames are to be encoded.

3. The method of claim 1 wherein the first order is a captured order as the frames were captured by an image capturing device and the second order is a display order as the frames are to be displayed.

4. The method of claim 1 wherein the first order is a coding order to decode the frames and the second order is a display order as the frames are to be displayed.

5. The method of claim 1 wherein I-frames and B-frames are provided to off-chip memory while P-frames are provided to on-chip memory rather than off-chip memory in order to be re-ordered to the second order in the second video sequence.

6. The method of claim 1 wherein I-frames, B-frames, and P-frames are provided to both on-chip memory and off-chip memory so that frame copies of the same frame from both on-chip and off-chip memories are re-ordered to the second order in the second video sequence.

7. The method of claim 1 wherein I-frames and P-frames are provided to off-chip memory while B-frames are provided to on-chip memory rather than off-chip memory in order to be re-ordered to the second order in the second video sequence.

8. The method of claim 1 wherein I-frames and P-frames are provided to off-chip memory while B-frames are provided to both on-chip memory and off-chip memory so that B-frame copies of the same frame from both on-chip and off-chip memories are re-ordered to the second order in the second video sequence.

9. The method of claim 1 wherein the local frames and copies of the local frames in an off-chip memory are both placed in the second video sequence in the second order.

10. The method of claim 1 comprising buffering some frames of a video sequence at off-chip memory and other frames of the video sequence at on-chip memory rather than off-chip memory or in addition to off-chip memory after scaling, formatting, and/or enhancing the image data of the frames; and
re-forming the video sequence with the frames of the off-chip and on-chip memory in display order.

11. A computer-implemented system comprising:
at least one display;
at least one on-chip memory and at least one off-chip memory to receive frames of image data;
at least one processor communicatively coupled to the memories and display,
and arranged to:
receive video frames of image data of a first video sequence having frames in a first order, wherein each video frame is a separate image;
perform a processing operation that modifies image data of the video frames to form local video frames;
directly after the processing operation, place the local video frames of the first video sequence into an on-chip cache memory, said on-chip cache memory with a capacity of a region of a single frame at a time, wherein said region is less than an entire frame, and without first obtaining the local video frames from an off-chip memory;
receive other frames of the first video sequence from the off-chip memory;
re-order the frames into a second video sequence having a second order different from the first order and comprising obtaining the local video frames via the on-chip memory without use of the off-chip memory and placing the local frames in the second video sequence according to the second order and with frames from the off-chip memory;
provide the frames in the second video sequence to code or display the image data; and
wherein the first video sequence order is associated with a first frame rate and the second video sequence order is associated with a second frame rate different from the first frame rate, and wherein the at least one processor being arranged to operate by placing both one frame from on-chip memory and a copy of the same frame from off-chip memory respectively in two consecutive frame spaces of the second video sequence order.

12. The system of claim 11 wherein the first order is a captured order as the frames were captured by an image capturing device and the second order is a coding order of frames as the frames are to be encoded.

13. The system of claim 11 wherein the first order is a captured order as the frames were captured by an image capturing device and the second order is a display order as the frames are to be displayed.

14. The system of claim 11 wherein the first order is a coding order to decode the frames and the second order is a display order as the frames are to be displayed.

15. The system of claim 11 wherein I-frames and B-frames are provided to off-chip memory while P-frames are provided to on-chip memory rather than off-chip memory in order to be re-ordered to the second order in the second video sequence.

16. The system of claim 11 wherein I-frames, B-frames, and P-frames are provided to both on-chip memory and off-chip memory so that frame copies of the same frame from both on-chip and off-chip memories are re-ordered to the second order in the second video sequence.

17. The system of claim 11 wherein I-frames and P-frames are provided to off-chip memory while B-frames are provided to on-chip memory rather than off-chip memory in order to be re-ordered to the second order in the second video sequence.

18. At least one non-transitory computer-readable medium having stored thereon instructions that when executed cause a computing device to:
receive video frames of image data of a first video sequence having frames in a first order, wherein each video frame is a separate image;
perform a processing operation that modifies image data of the video frames to form local video frames;
directly after the processing operation, place the local video frames of the first video sequence into an on-chip cache memory, said on-chip cache memory with a capacity of a region of a single frame at a time, wherein said region is less than a

Assignees

• Intel Corp

Inventors

• Sethuraman Ramanathan
• Mohan Sumit
• Wang Changliang
• Jiang Hong
• Giacalone Jean-Pierre

Classifications

• H04N19/433

  characterised by techniques for memory access · CPC title

• H04N19/172

  the region being a picture, frame or field · CPC title

• H04N19/127

  Prioritisation of hardware or computational resources · CPC title

• H04N19/159 Primary

  Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction · CPC title

• H04N19/31

  in the temporal domain · CPC title