Streaming application environment with recovery of lost or delayed input events

Patent metadata

Field	Value
Publication number	US-11366586-B2
Application number	US-201615368531-A
Country	US
Kind code	B2
Filing date	Dec 2, 2016
Priority date	Nov 18, 2016
Publication date	Jun 21, 2022
Grant date	Jun 21, 2022

Abstract

In a streaming application environment coupled to a remote device over a packet-switching network, correction events may be synthesized from input events received from the remote device and injected into a virtual streaming application to account for lost or delayed input events. In addition, trailing events may be reissued by a remote device in frames during which no new input events are generated.

First claim

What is claimed is:

1. A method, comprising, in a streaming application environment including one or more processors:
interfacing a virtual streaming application with a remote device coupled to the streaming application environment over a packet-switched network, including:
receiving, from the packet-switched network, input generated at the remote device, and
streaming an output display representation generated by the virtual streaming application in response to the received input over the packet-switched network to the remote device, the output display representation comprising a plurality of frames;

receiving a plurality of input events from the remote device, each of the input events, of the plurality of input events, being associated with an input generated on the remote device during rendering of the frames of the output display representation, and each input event, of the plurality of input events, being associated with a corresponding timestamp;
synthesizing at least one correction event from the plurality of received input events, the at least one correction event corresponding to a lost or delayed input event, of the plurality of input events, generated at the remote device during the rendering of the frames of the output display representation, the at least one correction event associated with a correction event timestamp for the at least one correction event, and in response to the input generated on the remote device;
generating a priority queue, of the plurality of input events and the at least one correction event, wherein generating the priority queue is based on the corresponding timestamps of the plurality of received input events and based on the correction event timestamp for the at least one correction event; and
controlling injection of the plurality of received input events and the at least one correction event into the virtual streaming application to synchronize, within the virtual streaming application and according to the corresponding timestamp for each input event in the priority queue and the correction event timestamp for the at least one correction event included in the priority queue, the plurality of received input events and the at least one correction event.

2. The method of claim 1 , wherein the plurality of received input events include gesture start, gesture stop, and gesture move events associated with user gestures received by the remote device, the method further comprising tracking a state of the user gestures.

3. The method of claim 2 , wherein the user gestures are associated with touching of a touchscreen, wherein the gesture start event is a touch down event, the gesture stop event is a touch up event, and the gesture move event is a touch move event.

4. The method of claim 2 , wherein tracking the state of the user gestures includes determining whether the gesture is started or completed.

5. The method of claim 4 , wherein synthesizing the at least one correction event corresponding to lost or delayed input event includes, in response to detecting a gesture start event when tracking of the state of the user gestures indicates that the gesture is started, synthesizing a gesture stop event to be injected prior to the detected gesture start event.

6. The method of claim 5 , wherein synthesizing the gesture stop event includes generating a duplicated event from the detected gesture start event, setting an action for the duplicated event to stop, setting coordinates for the duplicated event with last injected coordinates, and setting a timestamp for the duplicated event to a last gesture started timestamp.

7. The method of claim 4 , wherein synthesizing the at least one correction event corresponding to lost or delayed input event includes, in response to detecting a gesture stop event or a gesture move event when tracking of the state of the user gestures indicates that the gesture is completed, synthesizing a gesture start event to be injected prior to the detected gesture stop event or gesture move event.

8. The method of claim 7 , wherein synthesizing the gesture start event includes generating a duplicated event from the detected gesture stop event or gesture move event and setting an action for the duplicated event to start.

9. The method of claim 4 , further comprising tracking a last gesture started timestamp, wherein synthesizing the at least one correction event corresponding to lost or delayed input event includes, in response to detecting a gesture stop event or a gesture move event when tracking of the state of the user gestures indicates that the gesture is started and detecting that the corresponding timestamp of a matching gesture start event is different from the last gesture started timestamp, synthesizing a pair of gesture stop and gesture start events to be injected prior to the detected gesture stop event or gesture move event.

10. The method of claim 2 , further comprising synthesizing at least one companion gesture move event for injection following a received gesture start event.

11. The method of claim 2 , further comprising synthesizing at least one companion gesture move event for injection preceding a received gesture stop event.

12. The method of claim 2 , further comprising:
receiving, from the remote device, a trailing event generated during a frame of the remote device during which no gesture start, gesture stop, or gesture move events are generated; and
injecting the trailing event into the virtual streaming application as a last event of a virtual frame corresponding to the frame of the remote device in response to determining that a timestamp of the trailing event is later than the corresponding timestamp of a last input event injected into the virtual streaming application.

13. The method of claim 1 , wherein synthesizing the at least one correction event from at least one of the plurality of input events comprises:
inferring the at least one correction event based on at least a first input event, of the plurality of received input events, that is injected into the virtual streaming application on schedule according to the priority queue.

14. A system comprising one or more processors and memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to:
interface a virtual streaming application with a remote device coupled to the streaming application environment over a packet-switched network, including:
receiving from the packet-switched network input generated at the remote device, and
streaming an output display representation generated by the virtual streaming application in response to the received input over the packet-switched network to the remote device, the output display representation comprising a plurality of frames;

receive a plurality of input events from the remote device, each of the input events, of the plurality of input events, being associated with an input generated on the remote device during rendering of the frames of the output display representation, and each input event, of the plurality of input events, being associated with a corresponding timestamp;
synthesize at least one correction event from the plurality of received input events, the at least one correction event corresponding to a lost or delayed input event, of the plurality of input events, generated at the remote device during the rendering of the frames of the output display representation, the at least one correction event associated with a correction event timestamp for the at least one correction event, and in response

Assignees

• Google Inc
• Google Llc

Inventors

• Chang Chuo-Ling
• Tan Peter

Classifications

• H04N21/42224

  Touch pad or touch panel provided on the remote control · CPC title

• H04M1/72403

  with means for local support of applications that increase the functionality · CPC title

• G06F2203/0383

  Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN · CPC title

• G06F9/452

  Remote windowing, e.g. X-Window System, desktop virtualisation (protocols for virtual reality H04L67/131) · CPC title

• G06F9/451 Primary

  Execution arrangements for user interfaces · CPC title