Remoting desktop displays using move regions

Patent metadata

Field	Value
Publication number	US-10031636-B2
Application number	US-201113227491-A
Country	US
Kind code	B2
Filing date	Sep 8, 2011
Priority date	Sep 8, 2011
Publication date	Jul 24, 2018
Grant date	Jul 24, 2018

Abstract

A desktop display over a remote connection is updated by detecting, at their origin, changes in the desktop that reflect movement of an image, such as window scrolls, window moves and other movements of regions. For every frame of the desktop that is composed, moves can be detected. The offsets and regions of these moves are transformed into desktop space. A list of moves and other information about changed regions in the desktop display provide both an indication of moved image regions and move parameters, and image data that has otherwise changed. The moved image regions are moved at the destination by applying the move parameters, without transferring image data for the moved region to the remote device, while the remaining changed portion of the desktop is copied over the connection to the remote device.

First claim

What is claimed is:

1. A computer-implemented process, performed at a server computer, comprising:
rendering, in memory of the server computer, a desktop composition tree representing display data for a desktop display including at least one surface for an application and a desktop;
processing the desktop composition tree to identify changes in the desktop composition tree resulting in changes in the display data for the desktop display from a first frame to a next frame in a sequence of frames produced for the desktop display over time;
storing the identified changes in a plurality of move arrays, including at least one move array for the at least one surface and a move array for the desktop, the plurality of move arrays storing any move operation involving movement of a region from one position in the desktop display to another position in the desktop display, and further storing dirty regions for the at least one surface and dirty regions for the desktop;
generating, in the memory of the server computer, based on the identified changes, information describing moved regions of the desktop display and move parameters, including for each moved region an indication of a respective moved region and respective parameters of a move operation applied to the respective moved region, the respective parameters including information indicative of an offset of the respective moved region between the first frame and the next frame, and information describing dirty regions of the desktop display, including changed image data for the desktop display other than in the moved regions, by merging the at least one move array for the at least one surface and the move array for the desktop, including merging the dirty regions for the at least one surface and the dirty regions for the desktop; and
sending over a computer network to a remote display device, as display data for the the next frame, the information describing the moved regions of the desktop display and the move parameters, without image data of the desktop display for the moved regions, and the information describing the dirty regions of the desktop display, including the changed image data for the desktop display other than in the moved regions.

2. The computer-implemented process of claim 1 , wherein the generating comprises, for each rendering of the desktop, merging the information describing the moved regions of the desktop display and the move parameters with information from a previous rendering of the desktop, wherein the desktop is rendered again prior to sending the information from the previous rendering of the desktop.

3. The computer-implemented process of claim 1 , wherein processing the desktop composition tree to identify changes comprises detecting scroll operations applied to the at least one surface, and wherein generating the information describing the moved regions of the desktop display and the move parameters is based on at least the detected scroll operations.

4. The computer-implemented process of claim 1 , wherein the desktop composition tree further comprises a plurality of surfaces, wherein each surface of the plurality of surfaces represents an application, and each surface of plurality of surfaces has a respective move array and respective dirty regions.

5. The computer-implemented process of claim 1 , wherein the generating comprises, for each rendering of the at least one surface, merging the information describing the moved regions of the desktop display and the move parameters with information from a previous rendering of the at least one surface, wherein the at least one surface is rendered again prior to rendering the desktop display.

6. The computer-implemented process of claim 5 , wherein the merging the information describing the moved regions of the desktop display and the move parameters with the information from the previous rendering of the at least one surface comprises:
unioning new dirty regions with existing dirty regions;
for each new dirty region:
in response to a determination that there is a first intersection between a respective new dirty region and a source of a moved region, removing the first intersection from the source of the moved region; and
in response to a determination that there is a second intersection between the respective new dirty region and a destination of the moved region, removing, from the source of the moved region, the second intersection as translated back to the source of the moved region by an offset from move parameters for the moved region.

7. The computer-implemented process of claim 1 , wherein the sending over a computer network to the remote display device is accomplished without transferring pixel data for the moved regions of the desktop display to the remote display device.

8. A server computer comprising:
a processor and memory;
a computer storage medium and computer program instructions stored on the computer storage medium which, when processed by the processor, instruct the processor to perform a process comprising:
rendering, in the memory, a desktop composition tree representing display data for a desktop display including at least one surface for an application and a desktop;
processing the desktop composition tree to identify changes in the desktop composition tree resulting in changes in the display data for the desktop display from a first frame to a next frame in a sequence of frames produced for the desktop display over time;
storing, in the memory, the identified changes in a plurality of move arrays, including at least one move array for the at least one surface and a move array for the desktop, the plurality of move arrays storing any move operation involving movement of a region from one position in the desktop display to another position in the desktop display, and further storing dirty regions for the at least one surface and dirty regions for the desktop;
generating, in the memory, based on the identified changes, information describing moved regions of the desktop display and move parameters, including for each moved region an indication of a respective moved region and respective parameters of a move operation applied to the respective moved region, the respective parameters including information indicative of an offset of the respective moved region between the first frame and the next frame, and information describing dirty regions of the desktop display, including changed image data of the desktop display other than in the moved regions, by merging the at least one move array for the at least one surface and the move array for the desktop, including merging the dirty regions for the at least one surface and the dirty regions for the desktop; and
sending over a computer network to a remote display device, as display data for the the next frame, the information describing the moved regions of the desktop display and the move parameters, without image data of the desktop display for the moved regions, and the information describing the dirty regions of the desktop display, including the changed image data of the desktop display other than in the moved regions.

9. The server computer of claim 8 , wherein the generating comprises, for each rendering of the desktop, merging the information describing the moved regions of the desktop display and the move parameters with information from a previous rendering of the desktop, wherein the desktop is rendered again prior to sending the information from the previous rendering of the desktop.

10. The server computer of claim 8 , wherein processing the desktop composition tree to identify changes comprises detecting scroll operations applied to the at least one surface, and wherein generating the information describing the mov

Assignees

• Ungureanu Dorin
• Sadek Mohamed
• Schmieder Robert W
• Sidhana Harneet
• Microsoft Technology Licensing Llc

Inventors

• Ungureanu Dorin
• Sadek Mohamed
• Schmieder Robert W
• Sidhana Harneet

Classifications

• G06F3/048 Primary

  Interaction techniques based on graphical user interfaces [GUI] · CPC title

• G06F2203/0383

  Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN · CPC title