Video Decoding

Patent metadata

Field	Value
Publication number	US-2016234522-A1
Application number	US-201514615395-A
Country	US
Kind code	A1
Filing date	Feb 5, 2015
Priority date	Feb 5, 2015
Publication date	Aug 11, 2016
Grant date	—

Abstract

Video content is relayed from a transmitting device to a receiving device via a video relay server. The content comprises a plurality of frames. Before encoding, each of the plurality of frames is formed of a respective array of desired image data to be displayed at the receiving device. Filler image data is added to each of the plurality of frames before encoding. Control data is generated, which comprises cropping data indicating that none, or only some, of the filler image data should be cropped out before the plurality of frames is displayed. The encoded video content and the control data is transmitted to the server. At the server, the filler image data is detected automatically; in response, the cropping data is modified to indicate that all of the filler data should be cropped out before displaying the frames.

First claim

1 . A method for relaying video content from a transmitting device to a receiving device of a communication system, the method comprising:
at the transmitting device: receiving video content to be transmitted to the receiving device, the video content comprising a plurality of frames, each of the plurality of frames formed of a respective array of desired image data to be displayed at the receiving device; pre-processing the video content to add filler image data to each of the plurality of frames as more than a predetermined number of additional rows at the top and more than said predetermined number of additional rows at the bottom of the respective array; encoding the pre-processed video content; generating control data for decoding and displaying the plurality of frames, the control data comprising cropping data indicating that between zero and said predetermined number of topmost rows, inclusive, and between zero and said predetermined number of bottommost rows, inclusive, should be cropped out of each of the plurality of frames before the plurality of frames is displayed, thereby indicating that at least some of the additional rows should be displayed when the video content is outputted; and transmitting the encoded video content and the control data as a packet stream to a video relay server of the system; at the video relay server: receiving the packet stream; and executing stream processing code on a processor of the video relay server to: process at least part of the received packet stream to automatically detect the filler image data, in response to said detection, modify the packet stream by modifying the cropping data to indicate that all of the additional rows should be cropped out of each of the plurality of frames before the plurality of frames is displayed, and transmit the modified stream to the receiving device.

2 . A method according to claim 1 wherein the respective array of each of the plurality of frames has a total number of rows, wherein the pre-processing by the transmitting device further comprises reducing the resolution of the plurality of frames by at least reducing the total number of rows, wherein the addition of said more than said predetermined number of top rows and said more than said predetermined number of bottom rows is conditional on the reduced number being at or below a threshold, and wherein the reduced number is indicted in the control data by the transmitting device; and
wherein said processing by the stream processing code is of the control data in the packet stream to automatically detect the filler image data by detecting that the reduced number is at or below the threshold.

3 . A method according to claim 2 wherein the data stream is of data packets having headers containing payload data, wherein the control data is received as payload data contained in a control data packet, and wherein, for each packet in the received stream, the stream processing code determines from the header of that packet whether or not that packet is a control data packet and, if not, transmits that packet to the receiving device without modifying the payload data contained in that packet.

4 . A method according to claim 1 wherein the data stream is formatted according to the H.264 standard, HEVC standard, SMPTE VC-1 standard, or any other protocol which provides a Network Abstraction Layer (NAL) unit structure.

5 . A method according to claim 3 wherein the packets are NAL units, said control data packet is a sequence parameter set (SPS) NAL unit and, for each NAL unit in the received stream, the stream processing code determines from the header of that NAL unit whether or not that NALU unit is an SPS NAL unit and, if not, transmits that NAL unit to the receiving device without modifying the payload data contained in that NALU unit.

6 . A method according to claim 2 wherein the threshold is 288 rows.

7 . A method according to claim 1 wherein said modification by the stream processing code further comprises setting an aspect ratio display parameter in the control data to match the modified cropping data, thereby preventing disproportionate scaling of the plurality of video frames when displayed at the receiving device.

8 . A method according to claim 1 wherein the aspect ratio display parameter is set to substantially 16:9.

9 . A method according to claim 8 wherein the aspect ratio of each of the plurality of frames after pre-processing is substantially 11:9.

10 . A method according to claim 1 wherein the video content is call video of a call between a user of the receiving device and another user of the transmitting device, the packet stream being transmitted from the transmitting device to the relay server and modified by the stream processing code, and the modified stream being transmitted from the relay server to the receiving device, in real-time.

11 . A method according to claim 1 wherein the cropping data generated by the transmitting device is in the form of: a cropping flag set to a crop state, and a top and a bottom cropping parameter set to indicate that between one and said predetermined number of topmost rows, inclusive, and between one and said predetermined number of bottommost rows, inclusive, should be cropped out before the plurality of video frames is displayed respectively; and
wherein the cropping data is modified by setting the top and bottom cropping parameters to indicate that all of the additional rows should be cropped out before the plurality of video frames is displayed.

12 . A method according to claim 1 wherein the cropping data generated by the transmitting device is in the form of a cropping flag set to a non-crop state and thereby indicating that each of the plurality of frames, including the additional rows, should be displayed in its entirety when video content is outputted; and
wherein the cropping data is modified by setting the cropping flag to a crop state and adding a top and a bottom cropping parameter to the control data to indicate that all of the additional rows should be cropped out before the plurality of video frames is displayed.

13 . A method according to claim 1 further comprising the stream processing code decoding at least part of the video content from the received stream, the filler image data automatically detected by the stream processing code performing image recognition on the decoded at least part of the video content.

14 . A method according to claim 1 wherein said predetermined number is 15.

15 . A method according to claim 1 wherein the video content further comprises another plurality of frames, each of the other plurality of frames also formed of a respective array of desired image data to be displayed at the receiving device, wherein between zero and said predetermined number of further top rows of filler data, inclusive, and between zero and said predetermined number of further bottom rows of filler data, inclusive, is added to the other plurality of frames by the transmitting device when the video content is pre-processed, wherein the method further comprises:
generating at the transmitting device additional control data for decoding and displaying the other plurality of frames, the additional control data comprising additional cropping data indicating that all of the further top and bottom rows should be cropped out of the additional video frames, the additional control data also being included in the data stream; and
the stream processing code processing at least another part of the received stream to detect that there is no filler data in the other plurality of video frames which the further

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Lu Mei-Hsuan
• Bendapudi Satya Sasikanth

Classifications

• H04N19/85

  using pre-processing or post-processing specially adapted for video compression · CPC title

• H04N19/70

  characterised by syntax aspects related to video coding, e.g. related to compression standards · CPC title

• H04N19/463 Primary

  by compressing encoding parameters before transmission · CPC title

• H04N19/44

  Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder · CPC title

• H04N21/234372 Primary

  for performing aspect ratio conversion · CPC title