Presenting dynamically changing images in a limited rendering environment

Patent metadata

Field	Value
Publication number	US-9514242-B2
Application number	US-201213449076-A
Country	US
Kind code	B2
Filing date	Apr 17, 2012
Priority date	Aug 29, 2011
Publication date	Dec 6, 2016
Grant date	Dec 6, 2016

Abstract

Methods, systems, and techniques for presenting dynamically changing images in a limited rendering environment are described. Example embodiments provide a client display manager that is configured to receive image blocks representing modifications or updates to a dynamically changing image. The client display manager may then layer the received image blocks upon one another, and then cause the layered image blocks to be presented on a display device as a single, composited image. In some embodiments, multiple image blocks may be coalesced or otherwise combined into a single image transmitted to the client display manager, where regions of the single image that are not occupied by the multiple image blocks are transparent.

First claim

The invention claimed is:

1. A method for rendering a dynamically changing image in a limited display environment, the method comprising:
in a module of a web browser in a client device that has a display, the module not being configured to directly access a frame buffer associated with the display, presenting a dynamically changing image by,
receiving, a first hypertext transfer protocol (HTTP) message from a server that contains a first image block that includes data of the dynamically changing image of a user interface of a virtual machine (VM) being modified on the server at a first time;
generating a document object model (DOM) tree on the client device and adding the first image block as a first element in the DOM tree;
receiving a second HTTP message from the server that contains a second image block including data of the dynamically changing image of the user interface of the VM being modified on the server at a second time that occurs subsequently to the first time;
layering the second image block over the first image block in by adding to the DOM tree the second element corresponding to the second image block on the client;
causing the layered image blocks of the layering data structure to be rendered as a single image on the display;
determining that the layering data structure has reached its maximum capacity and cannot support additional layers; and
causing the server to transmit a new key frame image block and using the new key frame image block to clear the layering data structure on the client device.

2. The method of claim 1 wherein presenting the dynamically changing image includes presenting a user interface screen for code executing on a remote server system, and wherein the screen dynamically updates in response to user inputs.

3. The method of claim 1 , further comprising:
receiving a third image block representing two distinct modified portions of the dynamically changing image, the third image block including transparency in regions not occupied by the two distinct modified portions;
layering the third image block over previously received image blocks; and
causing the layered image blocks to be rendered as a single image on the display.

4. The method of claim 3 wherein the third image block including the transparency and representing the two distinct modified portions is received over a single HTTP network connection.

5. The method of claim 1 wherein the DOM tree is associated with an HTML document and accessed via a Cascading Style Sheet, wherein layering the second image block over the first image block includes assigning the first image block to a first z-index and assigning the second image block to a second z-index that is one higher than the first z-index, without copying either the first or the second image blocks into buffer images that represent all of the dynamically changing image.

6. The method of claim 1 wherein presenting the dynamically changing image is performed without use of double-buffering to display the first and second image blocks.

7. The method of claim 1 wherein presenting the dynamically changing image is performed without swapping buffer image layers.

8. The method of claim 1 wherein the module is JavaScript code executing within the Web browser executing on the client device.

9. The method of claim 1 wherein the client device includes a hardware frame buffer for the display, but the module accesses the display via an interface that does not provide any operations to copy image blocks into the hardware frame buffer.

10. The method of claim 1 wherein presenting the dynamically changing image includes presenting video transmitted from a remote server system to the client device.

11. The method of claim 1 , wherein the first image block is a key frame that represents an entire user interface rendered by the server, and wherein the second image block represents a modification to a portion of the key frame.

12. The method of claim 1 , wherein causing the server to transmit the new key frame image block further comprises:
determining the maximum capacity of the layering data structure by the client display manager; and
transmitting information indicating the maximum capacity to the server, wherein the server uses the information indicating the maximum capacity received from the client device in order to determine when to send new key frame image blocks to the client device.

13. The method of claim 1 , wherein causing the server to transmit the new key frame image block further comprises:
periodically detecting that the maximum capacity of the layering data structure has been reached on the client device; and
transmitting a request from the client device to the server, requesting the new key frame image block to be transmitted from the server to the client device.

14. A non-transitory computer-readable medium including contents that, when executed, enable a computing device that has a display to present a dynamically changing image using a limited display environment, by performing a method comprising:
in a module on a web browser of the computing device, the module not being configured to directly access a frame buffer associated with the display, presenting a dynamically changing image by,
receiving, a first hypertext transfer protocol (HTTP) message from a server that contains a first image block that includes data of the dynamically changing image of a user interface of a virtual machine (VM) being modified on the server at a first time;
generating a document object model (DOM) tree on the client device and adding the first image block as a first element in the DOM tree;
receiving a second HTTP message from the server that contains a second image block including data of the dynamically changing image of the user interface of the VM being modified on the server at a second time;
layering the second image block over the first image block in by adding to the DOM tree the second element corresponding to the second image block on the client;
causing the layered image blocks of the layering data structure to be rendered as a single image on the display, such that at least a portion of the first image block and a portion of the second image block become visible on the display;
determining that the layering data structure has reached its maximum capacity and cannot support additional layers; and
causing the server to transmit a new key frame image block and using the new key frame image block to clear the layering data structure on the client device.

15. The computer-readable medium of claim 14 wherein the computer-readable medium is a memory in the computing device.

16. The computer-readable medium of claim 14 wherein the contents are instructions that when executed cause the computing device to perform the method.

17. The computer-readable medium of claim 14 , wherein the method further comprises:
receiving a third image block representing two distinct modified portions of the dynamically changing image, the third image block including transparency in regions not occupied by the two distinct modified portions;
layering the third image block over previously received image blocks; and
causing the layered image blocks to be rendered as a single image on the display.

18. The computer-readable medium of claim 14 , wherein the first image block is a key frame that represents an entire user interface rendered by the server, and wherein the second image block represents a modification to a portion of the key frame.

19. The non-transitory computer-readable medium

Assignees

• Jooste Sarel Kobus
• Vmware Inc

Inventors

• Jooste Sarel Kobus

Classifications

• G09G5/14

  Display of multiple viewports · CPC title

• G06F9/4445

  Physics · mapped topic

• G09G2350/00

  Solving problems of bandwidth in display systems · CPC title

• G06F17/3028

  Physics · mapped topic

• H04L65/4084

  Electricity · mapped topic