User interface remoting through video encoding techniques

Patent metadata

Field	Value
Publication number	US-9277237-B2
Application number	US-201213561768-A
Country	US
Kind code	B2
Filing date	Jul 30, 2012
Priority date	Jul 30, 2012
Publication date	Mar 1, 2016
Grant date	Mar 1, 2016

Abstract

Methods, techniques, and systems for user interface remoting using video streaming techniques are provided. Example embodiments provide User Interface Remoting and Optimization System (“UIROS”), which enables the efficient remoting of pixel-oriented user interfaces on behalf of their guests using generic video streaming techniques, such as H.264, to send compressed user interface image information in the form of video frame encoded bitstreams. In one embodiment, the UIROS comprises server side support including a UI remoting server, a video encoder, and rendering support and client side support including a UI remoting client, a video decoder, and a display. These components cooperate to implement optimized UI remoting that is bandwidth efficient, low latency and CPU efficient.

First claim

The invention claimed is:

1. A method in a server computing system for remoting a user interface to be rendered on a client device, comprising:
during a motion estimation process of a generic video encoder, causing the generic video encoder to:
perform pattern matching on a graphics command stream of a quest operating system of a virtual machine on the server computing system in order to detect a move, scroll or expose region event;
when the move, scroll or expose region event is detected, determine, prior to causing an updated portion of the user interface to be rendered on a display screen associated with the client device, a set of pixels that can be reused in the updated portion of the user interface and identify a motion vector caused by the move, scroll or expose region event, wherein the motion vector indicates new locations for the set of pixels in the updated portion of the user interface;
using a generic video encoding protocol supported by a web browser of the client device, generate and encode a video-based representation of the updated portion of the user interface that indicates reuse of the determined set of pixels that can be reused in the updated portion of the user interface and the motion vector that indicates the new locations; and

sending the generated and encoded video-based representation of the updated portion of the user interface to the web browser on the client device in a manner that causes a decoder for the video encoding protocol on the web browser of the client device to decode and render the updated portion of the user interface on the display screen associated with the client device by using the set of pixels cached on the client device and the motion vector caused by the move, scroll or expose region event.

2. The method of claim 1 wherein the server computing system is a virtualization server computing system and the updated portion of the user interface is a portion of a desktop corresponding to a connection to a virtual machine.

3. The method of claim 1 wherein the server computing system is a physical computing system.

4. The method of claim 1 wherein the decoder for the video encoding protocol is a generic video decoder not optimized to decode video for user interface remoting.

5. The method of claim 1 wherein the determining a set of pixels that can be reused comprises analyzing pixels in a frame buffer of the server computing system to determine the set of pixels that can be reused in the updated portion of the user interface.

6. The method of claim 1 wherein determining a set of pixels that can be reused comprises reusing at least a portion of a video stream being streamed by the server computing system.

7. The method of claim 1 wherein the video encoding protocol is at least one of an H.264 protocol, H.265 protocol, VC1 protocol, or an VP8 protocol.

8. The method of claim 1 wherein the sending the generated video-based representation of the updated portion of the user interface to the client device further comprises sending the generated video-based representation of the updated portion of the user interface to a web browser executing on the client device in a manner that causes the web browser to forward the encoded representation to the decoder for the video encoding protocol.

9. The method of claim 8 , further comprising utilizing a <video> tag of the web browser to forward the encoded representation to the decoder for the video encoding protocol.

10. The method of claim 1 , further comprising sending the generated video-based representation of the updated portion of the user interface using TCP/IP.

11. The method of claim 1 wherein the sending the generated video-based representation of the updated portion of the user interface to the client device further comprises sending the generated video-based representation of the updated portion of the user interface directly to the decoder for the video encoding protocol.

12. The method of claim 1 wherein the method is performed by a virtual machine process and/or a virtual machine monitor of a virtualization server computing system.

13. The method of claim 1 , further comprising: detecting a latency in a network connecting the server computing system to the client device or in processing of encoded video-based representations by the client device; adjusting the encoded video-based representation to account for the latency.

14. The method of claim 13 wherein the adjusting comprises using less bandwidth.

15. The method of claim 13 wherein the adjusting comprises more compression of subsequent encoded video-based representations or slowing down the generating and sending of subsequent encoded video-based representations.

16. The method of claim 13 wherein the adjusting comprises minimizing CPU utilization on the client device.

17. The method of claim 13 wherein the adjusting comprises generating and sending fewer frames.

18. The method of claim 1 wherein the client device is at least one of a personal computer, a wireless or wirelessly connected device, a phone, a smart television, and/or a tablet.

19. The method of claim 1 , further comprising:
receiving a notification indicating user interface semantics that reuse pixel information from an application operating on the server computing system, wherein the application causes the user interface to be generated; and
generating the at least one predictive macroblock in response to receiving the notification received from the application.

20. The method of claim 1 , wherein the video encoder is further configured to receive a notification from a guest application on the guest operating system through an application programming interface (API), the notification indicating that the guest application has opportunistically pre-loaded pixel information on the client device using video decoder frame caching.

21. A non-transitory computer-readable medium stored in a server containing content for remoting a user interface to be rendered on a client device by performing a method comprising:
during a motion estimation process of a generic video encoder, causing the video encoder to:
perform pattern matching on a graphics command stream of a quest operating system of a virtual machine operating on the server computing system in order to detect a move, scroll or expose region event;
when the move, scroll or expose region event is detected, determine, prior to causing an updated portion of the user interface to be rendered on a display screen associated with the client device, pixels that can be reused in the updated portion of the user interface and identify a motion vector caused by the move, scroll or expose region event, wherein the motion vector indicates new locations for the pixels in the updated portion of the user interface;
using a generic video encoding protocol supported by a web browser of the client device, generate and encode a video-based representation of the updated portion of the user interface that indicates reuse of the determined pixels that can be reused in the updated portion of the user interface and the motion vector that indicates the new locations; and

sending the generated and encoded video-based representation of the updated portion of the user interface to the web browser on the client device in a manner that causes a decoder for the video encoding protocol on the web browser of the client device to decode and render the updated portion of the user interface on the display screen associated with the client device by using the pixels

Assignees

• Abiezzi Salim
• Whitwell Keith
• Vmware Inc

Inventors

• Abiezzi Salim
• Whitwell Keith

Classifications

• H04L65/70

  Media network packetisation · CPC title

• H04N19/51 Primary

  Motion estimation or motion compensation · CPC title

• H04N19/543 Primary

  using regions · CPC title

• H04N19/162

  User input · CPC title

• H04L69/04

  Protocols for data compression, e.g. ROHC · CPC title