WebRTC API redirection with screen sharing

Patent metadata

Field	Value
Publication number	US-10904325-B2
Application number	US-201816229308-A
Country	US
Kind code	B2
Filing date	Dec 21, 2018
Priority date	May 4, 2018
Publication date	Jan 26, 2021
Grant date	Jan 26, 2021

Abstract

A computing system includes a first client computing device, and a virtual desktop server. The first client computing device displays a local client surface, and a virtual client surface provided by the virtual desktop server. The virtual desktop server provides a real-time media application to provide real-time communications (RTC), and an API code redirection module to redirect intercepted APIs of the real-time media application intended for a native RTC engine based on redirection code injected into the real-time media application so that the portion of the real-time media application is redirected. The injected redirection code enumerates the local and virtual client surfaces. The first client computing device shares the local and virtual client surfaces with a second client computing device based on the intercepted APIs enumerating the local and virtual client surfaces.

First claim

That which is claimed:

1. A computing system comprising: a first client computing device configured to display a local client surface and a virtual client surface, with the local client surface and the virtual client surface to be shared with a second client computing device having peer-to-peer communications with the first client computing device;
a virtual desktop server comprising a processor configured to communicate with said first client computing device through a virtual channel to provide the virtual client surface, and comprising:
an application framework comprising a real-time media application to provide real-time communications (RTC), and a native RTC engine to execute a portion of the real-time media application when received by the native RTC engine, and
an API code redirection module to redirect intercepted APIs of the real-time media application intended for the native RTC engine based on redirection code injected into the real-time media application so that the portion of the real-time media application is redirected, with the injected redirection code enumerating the local and virtual client surfaces; and

said first client computing device comprising a client RTC API engine communicating configured to communicate with the API code redirection module through the virtual channel to execute the redirected portion of the real-time media application, with the redirected portion of the real-time media application invoked by the intercepted APIs corresponding to real-time media processing being off-loaded to said first client computing device, and configured to share the local and virtual client surfaces with the second client computing device based on the intercepted APIs enumerating the local and virtual client surfaces, wherein the redirected portion of the real-time media application is optimized to the capabilities of said first client computing device.

2. The computing system according to claim 1 wherein the real-time media application is configured to provide to said first client computing device a single list of enumerated sources of shared content for the local and virtual client surfaces.

3. The computing system according to claim 2 wherein the sources of shared content comprise at least one of applications and their windows, monitors and desktops.

4. The computing system according to claim 1 wherein the virtual client surface comprises at least one window that is to be shared with the second client computing device, and wherein said virtual desktop server is configured to provide pixels of the at least one window to the second client computing device.

5. The computing system according to claim 4 wherein said virtual desktop server is configured to compress the pixels of the at least one window.

6. The computing system according to claim 1 wherein the real-time media application is configured to prompt a user of said first client computing device to confirm that the local and virtual client surfaces are to be shared.

7. The computing system according to claim 1 wherein the virtual client surface comprises a plurality of overlapping windows, and wherein said first client computing device is configured to obtain, based on selection of one of the plurality of overlapping windows, pixels of the one selected window from said virtual desktop server.

8. The computing system according to claim 1 wherein said virtual desktop server is configured to share a whole virtual desktop of said virtual desktop server by capturing pixels of the virtual desktop and delivering the captured pixels to the second client computing device.

9. The computing system according to claim 1 wherein the virtual client surface comprises at least one window that is to be shared with the second client computing device, and wherein said virtual desktop server is configured to compress pixels of the at least one window before sending to said first client computing device for sharing with the second client computing device.

10. A method for operating a computing system comprising a virtual desktop server further comprising an application framework and an API code redirection module, with the application framework further comprising a real-time media application and a native RTC engine, and a first client computing device further comprising a client RTC API engine,
operating the first client computing device to display a local client surface and a virtual client surface, with the local client surface and the virtual client surface to be shared with a second client computing device having peer-to-peer communications with the first client computing device;
operating the virtual desktop server to communicate with the first client computing device through a virtual channel to provide the virtual client surface;
providing real-time communications (RTC) based on operation of the real-time media application, with a portion of the real-time media application to be executed by the native RTC engine when received by the native RTC engine;
redirecting by the API code redirection module intercepted APIs of the real-time media application intended for the native RTC engine based on redirection code injected into the real-time media application so that the portion of the real-time media application is redirected, with the injected redirection code enumerating the local and virtual client surfaces; and
operating the client RTC API engine communicating with the API code redirection module through the virtual channel to execute the redirected portion of the real-time media application, with the redirected portion of the real-time media application invoked by the intercepted APIs corresponding to real-time media processing being off-loaded to said first client computing device, and to share the local and virtual client surfaces with the second client computing device based on the intercepted APIs enumerating the local and virtual client surfaces, wherein the redirected portion of the real-time media application is optimized to the capabilities of said first client computing device.

11. The method according to claim 10 wherein the real-time media application provides to said first client computing device a single list of enumerated sources of shared content for the local and virtual client surfaces.

12. The method according to claim 11 wherein the sources of shared content comprise at least one of applications and their windows, monitors and desktops.

13. The method according to claim 10 wherein the virtual client surface comprises at least one window that is to be shared with the second client computing device, with pixels of the shared window to be provided by the virtual desktop server to the second client computing device.

14. The method according to claim 13 wherein the virtual desktop server compresses the pixels of the at least one window.

15. The method according to claim 10 wherein the real-time media application prompts a user of the first client computing device to confirm that the local and virtual client surfaces are to be shared.

16. The method according to claim 10 wherein the virtual client surface comprises a plurality of overlapping windows, and if one of the overlapping windows is selected to be shared with the second client computing device, then pixels of the selected window are obtained from the virtual desktop server.

17. The method according to claim 10 wherein if a whole virtual desktop of said virtual desktop server is to be shared, then pixels of the virtual desktop are captured on the first client computing device to be delivered to the second client computing device.

Assignees

• Citrix Systems Inc

Inventors

• Vysotsky Vladimir
• Giebler Todd
• Momchilov Georgy
• Page James

Classifications

• H04L65/403

  Arrangements for multi-party communication, e.g. for conferences (data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15) · CPC title

• H04L67/104 Primary

  Peer-to-peer [P2P] networks · CPC title

• H04L61/2589 Primary

  over a relay server, e.g. traversal using relay for network address translation [TURN] · CPC title

• G06F9/452

  Remote windowing, e.g. X-Window System, desktop virtualisation (protocols for virtual reality H04L67/131) · CPC title

• H04L67/59

  Providing operational support to end devices by off-loading in the network or by emulation, e.g. when they are unavailable · CPC title