WebRTC API redirection with window monitoring/overlay detection

Patent metadata

Field	Value
Publication number	US-10673939-B2
Application number	US-201816228964-A
Country	US
Kind code	B2
Filing date	Dec 21, 2018
Priority date	May 4, 2018
Publication date	Jun 2, 2020
Grant date	Jun 2, 2020

Abstract

A computing system includes at least one video source, a virtual desktop server and a client computing device. The virtual desktop server includes a real-time media application to provide real-time communications (RTC), a native RTC engine, a geometry tracking module, and an API code redirection module to redirect intercepted APIs of the real-time media application intended for the native RTC engine based on redirection code injected into the real-time media application. The injected redirection code defines a placeholder to indicate positioning geometry of a video stream within an RTC window. The geometry tracking module detects the placeholder within the injected redirection code. The client computing device includes a display composition module to receive the video stream and the positioning geometry of the placeholder, and to overlay the video stream over the placeholder within the displayed RTC window based on the positioning geometry.

First claim

That which is claimed:

1. A computing system comprising:
at least one video source to provide at least one video stream;
a virtual desktop server comprising:
an application framework comprising a real-time media application to provide real-time communications (RTC), and a native RTC engine to execute a portion of the real-time media application when received by the native RTC engine,
an API code redirection module to redirect intercepted APIs of the real-time media application intended for the native RTC engine based on redirection code injected into the real-time media application so that the portion of the real-time media application is redirected, with the injected redirection code defining at least one placeholder to indicate positioning geometry of the at least one video stream within an RTC window, and
a geometry tracking module to detect the at least one placeholder within the injected redirection code, and to provide the positioning geometry associated therewith; and

a client computing device comprising
a display to display the RTC window,
a client RTC API engine communicating with the API code redirection module through a virtual channel to execute the redirected portion of the real-time media application, and
and a display composition module to receive the at least one video stream and the positioning geometry of the at least one placeholder, and to overlay the at least one video stream over the at least one placeholder within the displayed RTC window based on the positioning geometry.

2. The computing system according to claim 1 wherein the geometry tracking module provides the positioning geometry to be included in the redirected portion of the real-time media application to the client RTC API engine, with the display composition module receiving the positioning geometry from the client RTC API engine.

3. The computing system according to claim 1 wherein the geometry tracking module provides the positioning geometry directly to the display composition module over a different virtual channel.

4. The computing system according to claim 1 wherein the displayed RTC window includes a composited display of non-accelerated graphics and accelerated graphics, with the accelerated graphics defined by the video stream.

5. The computing system according to claim 4 wherein the injected redirection code only generates the non-accelerated graphics, and wherein the geometry tracking module analyzes the non-accelerated graphics for the at least one placeholder.

6. The computing system according to claim 1 wherein the injected redirection code defines the at least one placeholder with at least one color, and wherein the geometry tracking module uses the at least one color to detect the at least one placeholder.

7. The computing system according to claim 6 wherein the injected redirection code changes the at least one color to at least one different color after detection by the geometry tracking module so that the at least one placeholder is displayed on the RTC window is in the at least one different color.

8. The computing system according to claim 1 wherein the injected redirection code defines the at least one placeholder with at least one pattern, and wherein the geometry tracking module uses the at least one pattern to detect the at least one placeholder.

9. The computing system according to claim 1 wherein the injected redirection code defines the at least one placeholder with at least one bar code, and wherein the geometry tracking module uses the at least one bar code to detect the at least one placeholder.

10. The computing system according to claim 9 wherein the at least one bar code assigns at least one ID for the at least one video stream.

11. The computing system according to claim 1 wherein the injected redirection code assigns at least one ID for the at least one placeholder that corresponds to the at least one video stream, and wherein the geometry tracking module associates the at least one ID with the at least one video stream.

12. The computing system according to claim 1 wherein the at least one video source comprises a camera coupled to said client computing device.

13. The computing system according to claim 1 wherein the at least one video source comprises a camera coupled to a different client computing device in peer-to-peer communications with said client computing device.

14. A method for operating a computing system comprising at least one video source, a virtual desktop server and a client computing device comprising a client RTC API engine, a display and a display composition module, with the virtual desktop server comprising a geometry tracking module, an application framework and an API code redirection module, and with the application framework comprising a real-time media application and a native RTC engine, the method comprising:
providing at least one video stream from the at least one video source;
providing real-time communications (RTC) based on operation of the real-time media application, with a portion of the real-time media application to be executed by the native RTC engine when received by the native RTC engine;
redirecting by the API code redirection module intercepted APIs of the real-time media application intended for the native RTC engine based on redirection code injected into the real-time media application so that the portion of the real-time media application is redirected, with the injected redirection code defining at least one placeholder to indicate positioning geometry of the at least one video stream within an RTC window;
operating the geometry tracking module to detect the at least one placeholder within the injected redirection code, and to provide the positioning geometry associated therewith;
displaying the RTC window on the display;
operating the client RTC API engine communicating with the API code redirection module through a virtual channel to execute the redirected portion of the real-time media application; and
operating the display composition module to receive the at least one video stream and the positioning geometry of the at least one placeholder, and to overlay the at least one video stream over the at least one placeholder within the displayed RTC window based on the positioning geometry.

15. The method according to claim 14 wherein the displayed RTC window includes a composited display of non-accelerated graphics and accelerated graphics, with the accelerated graphics defined by the video stream.

16. The method according to claim 15 wherein the injected redirection code only generates the non-accelerated graphics, and wherein the geometry tracking module analyzes the non-accelerated graphics for the at least one placeholder.

17. The method according to claim 14 wherein the injected redirection code defines the at least one placeholder with at least one color, and wherein the geometry tracking module uses the at least one color to detect the at least one placeholder.

18. The method according to claim 14 wherein the injected redirection code defines the at least one placeholder with at least one pattern, and wherein the geometry tracking module uses the at least one pattern to detect the at least one placeholder.

19. The method according to claim 14 wherein the injected redirection code defines the at least one placeholder with at least one bar code, and wherein the geometry tracking module uses the at least one bar code to detect the at least one placeholder.

20. A non-transitory computer readable medium for operating a vir

Assignees

• Citrix Systems Inc

Inventors

• Vysotsky Vladimir
• Giebler Todd
• Momchilov Georgy
• Page James

Classifications

• G06T7/90

  Determination of colour characteristics · CPC title

• H04L65/601

  Electricity · mapped topic

• H04L65/4015

  where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference · CPC title

• H04L65/608

  Electricity · mapped topic

• G06F9/45529

  Embedded in an application, e.g. JavaScript in a Web browser · CPC title