Video client optimization during pause

Patent metadata

Field	Value
Publication number	US-11758103-B2
Application number	US-202217949552-A
Country	US
Kind code	B2
Filing date	Sep 21, 2022
Priority date	Oct 1, 2018
Publication date	Sep 12, 2023
Grant date	Sep 12, 2023

Abstract

A system and method for providing quality control in immersive video during pausing of a video streaming session. In one embodiment, a paused video frame may comprise a plurality of mixed quality video tiles depending on user gaze vector information. Under pause control, the video quality of all tiles of the paused video frame is equalized such that it is of same value for all the video tiles, which may be the video quality of the tiles presented in a viewport of the client device. The paused video frame having the same quality tiles throughout is used as a replacement video frame, which is presented to the client device player for decoding and displaying instead of the mixed quality video frame while the streaming session is paused.

First claim

The invention claimed is:

1. A client device, comprising:
a video display;
at least one processor; and
a non-transitory computer-readable storage medium, coupled to the at least one processor, further including computer-readable instructions, when executed by the at least one processor, are configured to:
store, in a video decode buffer, one or more video frames of a video program downloaded in an ongoing immersive video session;
responsive to a user input, pause the ongoing immersive video session at a video frame of the video program, wherein the paused video frame is comprised of a first region having a first video quality and a second region having a second video quality;
equalize video quality of the paused video frame, wherein the video quality of the first region and video quality of the second region are of a same video quality;
present the paused video frame having the first and second regions of the same video quality to a media player for displaying the paused video frame using the video display while the immersive video session is in pause;
responsive to a determination the video decode buffer includes one or more video frames subsequent to the paused video frame of the immersive video session, flush the one or more subsequent video frames from the video decode buffer; and
fill the video decode buffer with one or more subsequent replacement video frames, wherein each subsequent replacement video frame comprises an entire video frame having a video quality at least equal to the equalized video quality of the paused video frame.

2. The client device as recited in claim 1 , wherein the same video quality of the paused video frame corresponds to the first video quality of the first region.

3. The client device as recited in claim 2 , wherein the first region of the paused video frame corresponds to a viewport of the video display defined by gaze vector information associated with a user of the client device engaged in the ongoing immersive video session.

4. The client device as recited in claim 1 , wherein the computer-readable instructions further comprise instructions configured to effectuate downloading of additional replacement video frames into the video decode buffer until the video decode buffer is full or a playback resume request is received.

5. A video optimization method, comprising:
storing, in a video decode buffer of a client device, one or more video frames of a video program downloaded in an ongoing immersive video session;
responsive to a user input, pausing the ongoing immersive video session at a video frame of the video program, wherein the paused video frame is comprised of a first region having a first video quality and a second region having a second video quality;
equalizing video quality of the paused video frame, wherein the video quality of the first region and video quality of the second region are of a same video quality;
presenting the paused video frame having the first and second regions of the same video quality to a media player for displaying the paused video frame using the video display while the immersive video session is in pause;
responsive to a determination the video decode buffer includes one or more video frames subsequent to the paused video frame of the immersive video session, flushing the one or more subsequent video frames from the video decode buffer; and
filling the video decode buffer with one or more subsequent replacement video frames, wherein each subsequent replacement video frame comprises an entire video frame having a video quality at least equal to the equalized video quality of the paused video frame.

6. The method as recited in claim 5 , wherein the same video quality of the paused video frame corresponds to the first video quality of the first region.

7. The client method as recited in claim 6 , wherein the first region of the paused video frame corresponds to a viewport of a video display associated with the client device, the viewport defined by gaze vector information associated with a user of the client device engaged in the ongoing immersive video session.

8. The method as recited in claim 5 , further comprising downloading of additional replacement video frames into the video decode buffer until the video decode buffer is full or a playback resume request is received.

9. An apparatus, comprising:
an immersive video display device; and
a client device including:
a processor;
a media player operative to play a video program comprising a plurality of video frames;
a video decode buffer for storing one or more video frames of the video program in an ongoing immersive video session; and
a pause control module comprising executable program instructions for performing following acts when executed by the processor of the client device:
receiving a request to pause the ongoing video session at a video frame comprising an in-view portion and an out-of-view portion, the in-view portion corresponding to a viewport associated with the immersive video display device and comprising a first video quality, and the out-of-view portion comprising a region outside the viewport and comprising a second video quality less than the first video quality;
generating a replacement frame corresponding to the video frame paused, the replacement frame comprising a uniform video quality at least equal to the first video quality;
presenting the replacement frame to the media player for display during pausing; and
while pausing, flushing a pause buffer associated with the client device and obtaining high quality video frames subsequent to the video frame being paused for storing in the pause buffer associated with the client device, the high quality video frames having a video quality at least equal to the first video quality.

10. The apparatus as recited in claim 9 , wherein the pause control module further comprises program instructions for performing:
determining that the video decode buffer contains one or more video frames subsequent to the video frame being paused, the one or more subsequent video frames each having a variable video quality across the video frame; and
responsive to the determining, flushing the one or more subsequent video frames from the video decode buffer.

11. The apparatus as recited in claim 10 , wherein the pause control module further comprises program instructions for filling the video decode buffer with one or more replacement frames subsequent to the video frame being paused, each subsequent replacement frame having a video quality at least equal to the first video quality.

12. The apparatus as recited in claim 9 , wherein the pause control module further comprises program instructions for continuing to download additional subsequent replacement video frames into the pause buffer until the pause buffer is full or a playback resume request is received.

13. The apparatus as recited in claim 9 , wherein the pause control module further comprises program instructions for performing, responsive to receiving a playback resume request, commencing playback of the video session from a subsequent replacement video frame immediately following the video frame being paused, the subsequent replacement video frame having a video quality at least equal to the first video quality regardless of gaze vector information obtained when the playback resume request is received.

14. A video optimization method, comprising:
receiving a request to pause an ongoing video session at a video frame comprising an in-view portion and an out-of-view portion, the in-view portion corresponding to a viewport and comprising a first video quality and the out-of-view portion

Assignees

• Ericsson Telefon Ab L M

Inventors

• Phillips Chris
• Forsman Robert Hammond
• Cohen Sarel

Classifications

• H04N19/174 Primary

  the region being a slice, e.g. a line of blocks or a group of blocks · CPC title

• H04N13/117 Primary

  the virtual viewpoint locations being selected by the viewers or determined by viewer tracking · CPC title

• H04N19/40

  using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream · CPC title

• H04N19/146

  Data rate or code amount at the encoder output · CPC title

• H04N19/597

  specially adapted for multi-view video sequence encoding · CPC title