Interactive latency measurement

Patent metadata

Field	Value
Publication number	US-11338199-B2
Application number	US-202016793282-A
Country	US
Kind code	B2
Filing date	Feb 18, 2020
Priority date	Feb 18, 2020
Publication date	May 24, 2022
Grant date	May 24, 2022

Abstract

The described technology is generally directed towards techniques to measure interaction latency. Interaction latency can be measured at a client device by measuring time intervals between user inputs and corresponding feedback, such as video frames responsive to the user inputs. Feedback can comprise communication bursts received at the client device. The communication bursts can be detected and correlated with user inputs in order to measure interaction latency. Feedback can also comprise video response features which are responsive to the user inputs. Received video frames can be analyzed to discover response features, and video frames including the response features can be correlated with user inputs in order to measure interaction latency.

First claim

What is claimed is:

1. A client device, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising:
sending an input to a remote video interactive application, wherein the input is sent via an input data packet;
receiving, from the remote video interactive application, response data packets comprising a video stream, wherein the video stream comprises multiple video frames;
processing the multiple video frames to extract a response feature, wherein the response feature comprises a response to the input;
identifying a video frame of the multiple video frames, the video frame comprising the response feature; and
to estimate an interaction latency between the client device and the remote video interactive application, measuring a time interval between the sending of the input data packet to the remote video interactive application and receiving of a response data packet comprising the video frame comprising the response feature.

2. The client device of claim 1 , wherein the processing the multiple video frames to extract the response feature comprises extracting multiple features from the multiple video frames, and performing a feature classification to identify the response feature among the multiple features.

3. The client device of claim 1 , wherein the response feature comprises at least one of a blur or a color change in at least a portion of the video frame.

4. The client device of claim 1 , wherein the input comprises an input to open a menu for the remote video interactive application, and wherein the response feature comprises the menu for the remote video interactive application.

5. The client device of claim 1 , wherein the operations further comprise sending a plurality of inputs to the remote video interactive application, and correlating the response feature with a respective input of the plurality of inputs.

6. The client device of claim 1 , wherein the operations further comprise reporting the time interval to a remote interaction latency reporting service according to a reporting protocol.

7. The client device of claim 1 , wherein the operations further comprise measuring a plurality of time intervals between sending inputs of a plurality of inputs to the remote video interactive application and receiving video frames of a plurality of video frames comprising response features, and combining the plurality of time intervals to estimate the interaction latency between the client device and the remote video interactive application.

8. The client device of claim 1 , wherein the sending the input to the remote video interactive application comprises sending the input to the remote video interactive application via a cellular communications network.

9. A method, comprising:
sending, by a client device comprising a processor, inputs to a remote interactive application, wherein the inputs are sent via input data packets;
receiving, by the client device, communication bursts, wherein a communication burst of the communication bursts comprises an increase in a rate of arrival of data packets at the client device;
correlating, by the client device, inputs of the inputs to communication bursts of the communication bursts, resulting in correlated inputs and communication bursts;
measuring, by the client device, time intervals between inputs and communication bursts of the correlated inputs and communication bursts; and
using, by the client device, the time intervals to estimate an interaction latency between the client device and the remote interactive application.

10. The method of claim 9 , wherein the remote interactive application comprises a video game application.

11. The method of claim 9 , wherein the communication burst of the communication bursts comprises an increased number of streaming video data packets received at the client device.

12. The method of claim 9 , wherein the client device receives the communication bursts from the remote interactive application via a network device of a cellular communications network.

13. The method of claim 9 , further comprising detecting, by the client device, a time interval of the time intervals that exceeds an estimated interaction latency between the client device and the remote interactive application.

14. The method of claim 13 , further comprising reporting, by the client device, the time interval to a remote device associated with a remote interaction latency reporting service.

15. A machine-readable storage medium, comprising executable instructions that, when executed by a processor at a server, facilitate performance of operations, comprising:
correlating, at the server, video game inputs with response features, each response feature of the response features comprising a feature of a video frame generated by a video game application in response to a video game input, the correlating resulting in correlated video game inputs and correlated response features;
deploying, from the server to a client device, correlation data that is representative of the correlated video game inputs and the correlated response features; and
receiving, at the server from the client device, client device interaction latency measurement data representative of time intervals between the client device sending video game inputs of the correlated video game inputs to the video game application and the client device receiving data packets comprising response features of the correlated response features from the video game application.

16. The machine-readable storage medium of claim 15 , wherein the correlating the video game inputs with response features comprises using a deep neural network to analyze the video game inputs and the response features.

17. The machine-readable storage medium of claim 15 , wherein the correlating the video game inputs further comprises correlating the video game inputs with communication bursts, each communication burst comprising an increased volume of video frames generated by the video game application, as compared to a background volume of video frames generated by the video game application.

18. The machine-readable storage medium of claim 15 , wherein the operations further comprise, in response to the receiving the client device interaction latency measurement data, changing a setting of a cellular communications network used for communication between the client device and the video game application.

19. The machine-readable storage medium of claim 15 , wherein a video game input of the video game inputs comprises a video game weapon activation, and wherein a response feature correlated with the video game weapon activation comprises a blur or a color change in at least a portion of the video frame generated by the video game application in response to the weapon activation.

20. The machine-readable storage medium of claim 15 , wherein a video game input of the video game inputs comprises an input to open a menu for the video game application, and wherein a response feature correlated with the input to open the menu comprises the menu for the video game application in at least a portion of the video frame generated by the video game application in response to the input to open the menu.

Assignees

• At & T Ip I Lp

Inventors

• Hwang Michael
• Tariq Sannan
• Rath Christopher
• Jana Rittwik
• Shi Shu

Classifications

• A63F13/335 Primary

  using Internet · CPC title

• A63F2300/534

  for network load management, e.g. bandwidth optimization, latency reduction · CPC title

• A63F2300/538

  for performing operations on behalf of the game client, e.g. rendering · CPC title

• A63F2300/535

  for monitoring, e.g. of user parameters, terminal parameters, application parameters, network parameters · CPC title

• A63F13/332

  using wireless networks, e.g. cellular phone networks · CPC title