Low latency screen mirroring

Patent metadata

Field	Value
Publication number	US-2016350056-A1
Application number	US-201514958336-A
Country	US
Kind code	A1
Filing date	Dec 3, 2015
Priority date	May 28, 2015
Publication date	Dec 1, 2016
Grant date	—

Abstract

Aspects of the present disclosure relate to graphics domain transmission methods that utilize an adaptive compression pipeline to achieve low latency screen mirroring between a source device and a sink device. A source device captures a plurality of graphics domain frames, each of the graphics domain frames including one or more graphics command tokens. The source device utilizes an adaptive compression pipeline to compress the graphics domain frames based on one or more characteristics of the frames, and the adaptive compression pipeline is configured to perform at least one of scalable texture streaming, frame-based prediction, frame dropping, or data compression. The source device transmits the compressed frames to a sink device, and displays a rendered image of the graphics domain frames in time synchronization with a corresponding rendered image of the compressed frames displayed at the sink device.

First claim

What is claimed is:

1 . A source device, comprising:
a communications interface; a memory storing executable code; and at least one processor communicatively coupled to the communications interface and the memory, wherein the at least one processor and the memory are configured to:
capture a plurality of graphics domain frames, each of the graphics domain frames comprising one or more graphics command tokens;
utilize an adaptive compression pipeline to compress the plurality of graphics domain frames based on one or more characteristics of the frames, wherein the adaptive compression pipeline is configured to perform at least one of scalable texture streaming, frame-based prediction, frame dropping, or data compression;
transmit the compressed frames via the communications interface to a sink device; and
display a rendered image of the graphics domain frames in time synchronization with a corresponding rendered image of the compressed frames displayed at the sink device.

2 . The source device of claim 1 , wherein, for frame dropping, the at least one processor and memory are further configured to drop a first frame of the plurality of graphics domain frames in connection with a predetermined condition comprising at least one of:
the first frame following a predetermined number of second frames leading the plurality of graphics domain frames; the first frame comprising no textures; the first frame and a previous frame of the first frame comprising a same command token; at least one frame of a predetermined number of previous frames of the first frame not being dropped; or a communications delay between the source device and the sink device being greater than a threshold duration.

3 . The source device of claim 1 , wherein, for scalable texture streaming, the at least one processor and memory are further configured to:
divide a first frame of the plurality of graphics domain frames into a plurality of stages, wherein the first frame comprises textures; and transmit the compressed frames comprising the plurality of stages in sequence to the sink device, wherein the first frame is configured to be rendered at the sink device by successive interpolation of the stages.

4 . The source device of claim 1 , wherein, for frame-based prediction, the at least one processor and memory are further configured to:
if two consecutive frames of the plurality of graphics domain frames are equal in length:
determine one or more same locations of the two consecutive frames where the same locations have different data;
encode the locations corresponding to the different data; and
transmit the compressed frames comprising the encoded locations and the different data to the sink device.

5 . The source device of claim 1 , wherein, if a first frame of the plurality of graphics domain frames comprises textures, the at least one processor and memory are further configured to:
separate a red green and blue (RGB) image from an alpha channel of the textures; compress the RGB image utilizing lossy compression; and selectively compress the alpha channel by lossy compression based on a content of the alpha channel.

6 . The source device of claim 1 , wherein the graphics command tokens comprise OpenGL (Open Graphics Library) command tokens.

7 . The source device of claim 1 , wherein the plurality of graphics domain frames comprise no rendered graphics.

8 . A sink device, comprising:
a communications interface; a memory comprising executable code; and at least one processor communicatively coupled to the communications interface and the memory, wherein the at least one processor and the memory are configured to:
receive a plurality of graphics domain frames from a source device via the communications interface,
wherein each of the graphics domain frames comprises one or more graphics command tokens, and the plurality of graphics domain frames are compressed by an adaptive compression pipeline comprising at least one of scalable texture streaming, frame-based prediction, frame dropping, or data compression; and
selectively display at least some of the plurality of graphics domain frames based on a timestamp of each of the plurality of frames.

9 . The sink device of claim 8 , wherein the at least one processor and the memory are further configured to refrain from displaying a first frame of the plurality of graphics domain frames if the first frame is received after the timestamp of the first frame.

10 . The sink device of claim 9 , wherein the at least one processor and the memory are further configured to render the first frame.

11 . The sink device of claim 8 , wherein the at least one processor and the memory are further configured to refrain from displaying a frame of the plurality of graphics domain frames if the frame arrives at the sink device after a timestamp of the frame.

12 . The sink device of claim 8 , wherein the one or more graphics command tokens comprise OpenGL (Open Graphics Library) command tokens.

13 . The sink device of claim 8 , wherein the plurality of graphics domain frames comprise no rendered graphics.

14 . A method of screen mirroring operable at a source device, comprising:
capturing a plurality of graphics domain frames of screen content, each of the frames comprising one or more graphics command tokens; utilizing an adaptive compression pipeline to compress the plurality of graphics domain frames based on one or more characteristics of the frames, wherein the adaptive compression pipeline is configured to perform at least one of scalable texture streaming, frame-based prediction, frame dropping, or data compression; transmitting the compressed frames via a communications interface of the source device to a sink device; and displaying a rendered image of the frames in time synchronization with a corresponding rendered image of the compressed frames displayed at the sink device.

15 . The method of claim 14 , wherein, for frame dropping, utilizing the adaptive compression pipeline comprises dropping a first frame of the plurality of graphics domain frames in connection with a predetermined condition comprising at least one of:
the first frame following a predetermined numbers of second frames leading the plurality of frames; the first frame comprising no textures; the first frame and a previous frame of the first frame comprising a same command token; at least one frame of a predetermined number of previous frames of the first frame not being dropped; or a communications delay between the source device and the sink device being greater than a threshold duration.

16 . The method of claim 14 , wherein, for scalable texture streaming, utilizing the adaptive compression pipeline comprises:
dividing a first frame of the plurality of graphics domain frames into a plurality of stages, wherein the first frame comprises textures; and transmitting the compressed frames comprising the plurality of stages in sequence to the sink device, wherein the first frame is rendered at the sink device by successive interpolation of the stages.

17 . The method of claim 14 , wherein, for frame-based prediction, utilizing the adaptive compression pipeline comprises:
if two consecutive frames of the plurality of graphics domain frames are equal in length:
determining one or more same locations of the two consecutive frames where the same locations have different data;
encoding the locations corresponding to the different data; and
transmitting the compressed frames comprising the encoded locations and the different

Assignees

• Qualcomm Inc

Inventors

• Makar Mina Ayman Saleh Yanni
• Wang Xiaodong
• Jiang Hongyu
• Yu Na
• Raveendran Vijayalakshmi

Classifications

• G06F3/1454 Primary

  involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay · CPC title

• G09G2340/02

  Handling of images in compressed format, e.g. JPEG, MPEG · CPC title

• H04L69/04

  Protocols for data compression, e.g. ROHC · CPC title

• G09G2310/08

  Details of timing specific for flat panels, other than clock recovery · CPC title

• G06F9/452

  Remote windowing, e.g. X-Window System, desktop virtualisation (protocols for virtual reality H04L67/131) · CPC title