Dynamic representation of remote computing environment

Patent metadata

Field	Value
Publication number	US-11245772-B1
Application number	US-201916369692-A
Country	US
Kind code	B1
Filing date	Mar 29, 2019
Priority date	Mar 29, 2019
Publication date	Feb 8, 2022
Grant date	Feb 8, 2022

Abstract

Systems and methods for dynamic representation of a remote computing environment which can be implemented in the context of a virtual desktop infrastructure. A server hosting a computing environment may communicate system state to a remote client via a network. A representation of the computing environment may be generated by obtaining graphical data of a virtual computing environment, sub sampling the graphical data to determine a plurality of regions, determining a manner to prioritize the plurality of regions, and causing, based on the prioritization, an approximate representation of the graphical data to be transmitted to a second system.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
determining graphical data of a virtual computing environment;
sub sampling the graphical data to determine a plurality of regions;
determining, for a region of the plurality of regions, metadata associated with a manner in which to prioritize data of the region;
determining an approximate representation of the virtual computing environment based on the metadata by at least omitting state information of one or more elements of the graphical data according to the determined manner in which to prioritize the data, wherein regions of the plurality of regions are independently parsable, wherein prioritization of the data includes instructions to prioritize a text region over an image region; and
transmitting the approximate representation as a plurality of messages to a client of the virtual computing environment.

2. The computer-implemented method of claim 1 , wherein determining metadata associated with the manner in which to prioritize the data of the region comprises analyzing the region based on at least one of: visual complexity, visual shape, time history, and user input.

3. The computer-implemented method of claim 1 , wherein obtaining the graphical data of the virtual computing environment comprises capturing one or more screenshots of a graphical user interface (GUI) data and sub sampling the graphical data to determine the plurality of regions comprises analyzing the screenshots to determine the plurality of regions.

4. The computer-implemented method of claim 1 , further comprising encoding the approximate representation according to a H.264 format.

5. The method of claim 1 , further comprising: as a result of the instructions to prioritize the static text region, buffering a video in the image region based on the determined manner in which to prioritize the data.

6. A system, comprising:
one or more processors; and
memory that stores computer-executable instructions executable to cause the one or more processors to:
determine graphical data of a virtual computing environment;
subsample the graphical data to determine a plurality of regions;
determine a manner to prioritize the plurality of regions; and
cause, based on the prioritization, an approximate representation of the graphical data to be transmitted to a second system, wherein one portion of the approximate representation is independently parsable from another portion, further wherein the approximate representation omits state information of one or more elements of the graphical data, wherein the prioritization of the plurality of regions includes prioritization of the one portion comprising a non-image region over the another portion comprising an image region.

7. The system of claim 6 , wherein the instructions to determine the manner to prioritize the plurality of regions include instructions that, if executed, cause the one or more processors to analyze a region based on at least one of: visual complexity, visual shape, time history, and user input.

8. The system of claim 6 , wherein the instructions to cause, based on the prioritization, the approximate representation of the graphical data to be transmitted to a second system includes instructions executable to cause the one or more processors to buffer a video region of the plurality based on the determined manner of prioritization.

9. The system of claim 8 , wherein the manner of prioritization is determined based on the video being on-demand video.

10. The system of claim 6 , wherein the instructions to subsample the graphical data to determine the plurality of regions include instructions executable to cause the one or more processors to determine a region of the plurality by at least using a camera to track a user's field of vision.

11. The system of claim 6 , wherein the system is implemented as part of a virtual desktop infrastructure.

12. The system of claim 6 , wherein the instructions to cause, based on the prioritization, the approximate representation of the graphical data to be transmitted to the second system include instructions that, as a result of execution, cause the system to transmit the approximate representation using User Datagram Protocol (UDP).

13. A non-transitory computer-readable storage medium comprising executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to at least:
subsample graphical data of a computing environment to determine a plurality of regions, wherein the graphical data comprises scrolling text data and image data;
determine a manner in which to prioritize the plurality of regions, wherein prioritization of a region of the plurality of regions is based on sub sampled graphical data of the region;
determine an approximate representation of the computing environment based on the plurality of regions and the manner of prioritization; and
transmit the approximate representation as a plurality of data packets to a second computer system, wherein a first data packet of the plurality of data comprising the scrolling text data packets is parsable in absence of a second data packet comprising the image data of the plurality of data packets by using placeholder image data in place of the image data.

14. The non-transitory computer-readable storage medium of claim 13 , wherein a first region of the plurality of regions comprising high contrast graphical data is prioritized over a second region of the plurality of regions lacking the high contrast graphical data.

15. The non-transitory computer-readable storage medium of claim 13 , wherein the plurality of data packets comprises a data packet encoding a video that is prioritized after other packets of the plurality of data packets, further wherein the approximate representation comprises video playback being buffered.

16. The non-transitory computer-readable storage medium of claim 13 , wherein the manner in which to prioritize the plurality of regions is based on whether there is user interaction in a particular region.

17. The non-transitory computer-readable storage medium of claim 13 , wherein the instructions further include instructions that, as a result of execution, further cause the computer system to:
detect a portion of the computing environment that includes video playback;
obtain an indication from a user of the second computer system that buffering of the video playback is acceptable; and
buffer the video playback as part of the approximate representation.

18. The non-transitory computer-readable storage medium of claim 13 , wherein the computer system is at least a part of a computing resource service provider and the second system is a client of the computing resource service provider.

19. The non-transitory computer-readable storage medium of claim 13 , wherein the plurality of data packets lacks data associated with unchanged regions of the computing environment.

20. The non-transitory computer-readable storage medium of claim 13 , wherein the approximate representation omits state information of one or more elements of the graphical data.

Assignees

• Amazon Tech Inc

Inventors

• Vilke Stephen Daniel
• Selvidge James Morgan
• Willis Rudy
• Hinks Paul

Classifications

• H04L67/5651 Primary

  Reducing the amount or size of exchanged application data · CPC title

• G06F2203/0383

  Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN · CPC title

• G06F3/038

  Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry · CPC title

• H04N19/37

  with arrangements for assigning different transmission priorities to video input data or to video coded data · CPC title

• H04N19/17

  the unit being an image region, e.g. an object · CPC title