Detecting latency anomalies from pipeline components in cloud-based systems

Patent metadata

Field	Value
Publication number	US-2021377623-A1
Application number	US-202016886192-A
Country	US
Kind code	A1
Filing date	May 28, 2020
Priority date	May 28, 2020
Publication date	Dec 2, 2021
Grant date	—

Abstract

A method, computer readable medium, and system are disclosed for monitoring a pipeline to detect anomalies such as unusual latency associated with a particular stage. Each stage of the pipeline is configured to update metadata associated with content being processed by inserting a time stamp into the metadata when processing of the content is completed by the stage. The server device can collect the metadata from the last stage of the pipeline and analyze the metadata in order to generate metrics for the pipeline, including a residual latency and/or a gain for each stage of the pipeline. In an embodiment, the content is a frame of video to be displayed on a client device after being rendered by a server device, such as through a streaming service (e.g., a video game streaming service). The server device can adjust the pipeline based on the metrics to improve performance.

First claim

What is claimed is:

1 . A system for monitoring latency variability in a pipeline, the system comprising:
a server device including one or more processors configured to:
receive, from a client device, metadata corresponding to content processed by a plurality of stages of the pipeline, wherein the metadata includes information corresponding to each stage of the plurality of stages of the pipeline;
generate, for each stage of the plurality of stages of the pipeline, a metric calculated based on the metadata; and
adjust a parameter associated with at least one stage of the pipeline based on the metrics for the plurality of stages.

2 . The system of claim 1 , wherein the content comprises a plurality of frames streamed to the client device during a session, and wherein the one or more processors are further configured to:
calculate, for each frame of the plurality of frames, a gain for each stage of the pipeline; compare, for each frame of the plurality of frames, a total residual latency to a first threshold value to identify anomalous frames; calculate, for each stage of the pipeline, an aggregate value by summing a number of anomalous frames in the plurality of frames where the gain corresponding to the stage in the anomalous frame is greater than a second threshold value; and generate a visual representation based on the metrics, wherein the metric for a particular stage is calculated based on the aggregate value for the particular stage.

3 . The system of claim 2 , wherein the gain for a particular stage is calculated based on an expected frame duration and a difference between a time stamp corresponding to the particular stage for the current frame and a time stamp corresponding to the particular stage for a previous frame.

4 . The system of claim 3 , wherein the pipeline generates the plurality of frames in accordance with a variable frame rate, and wherein the expected frame duration is set in accordance with a target frame rate.

5 . The system of claim 2 , wherein the metric for a particular stage is calculated by multiplying a rate of the gain for the particular stage exceeding the second threshold value by a mean of the gain for the particular stage over a duration of the session.

6 . The system of claim 1 , wherein the content comprises a frame of video rendered in response to a feedback signal received from the client device.

7 . The system of claim 6 , wherein the frame of video is rendered in accordance with a ray-tracing algorithm.

8 . The system of claim 6 , wherein the feedback signal includes at least one signal selected from the group consisting of:
controller input; or sensor data from a head-mounted display.

9 . The system of claim 1 , wherein adjusting the parameter causes an encoding stage or a decoding stage of the pipeline to use a different encoding configuration or a different codec for encoding or decoding the content.

10 . The system of claim 1 , wherein the information corresponding to each stage of the pipeline comprises a time stamp that indicates a time when the processing of the content by the stage is complete.

11 . The system of claim 1 , wherein each stage includes one or more components and at least one stage includes a plurality of components, and wherein information corresponding to a particular stage of the pipeline comprises a time stamp for each component of the particular stage that indicates a time when the processing of the content by the component is complete.

12 . The system of claim 1 , wherein the pipeline includes at least one stage selected from the group consisting of: a frame capture stage, an encoding stage, a forward error correction (FEC) stage, a transmit stage, a network stage, a decoding stage, a resynchronization stage, or a display stage.

13 . The system of claim 1 , wherein the client device is configured to train a neural network for an autonomous vehicle, and wherein the server device is operable to generate simulated camera signals, representative of camera signals captured by the autonomous vehicle, in response to control inputs received from the autonomous vehicle.

14 . A method, comprising:
receiving, from a client device, metadata corresponding to content processed by a plurality of stages of a pipeline, wherein the metadata includes information corresponding to each stage of the plurality of stages of the pipeline; generating, for each stage of the plurality of stages of the pipeline, a metric calculated based on the metadata; and adjusting a parameter associated with at least one stage of the pipeline based on the metrics for the plurality of stages.

15 . The method of claim 14 , wherein the content comprises a plurality of frames streamed to the client device during a session, the method further comprising:
calculating, for each frame of the plurality of frames, a gain for each stage of the pipeline; comparing, for each frame of the plurality of frames, a total residual latency to a first threshold value to identify anomalous frames; calculating, for each stage of the pipeline, an aggregate value by summing a number of anomalous frames in the plurality of frames where the gain corresponding to the stage in the anomalous frame is greater than a second threshold value; and generating a visual representation based on the metrics, wherein the metric for a particular stage is calculated based on the aggregate value for the particular stage.

16 . The method of claim 15 , wherein the gain for a particular stage is calculated based on an expected frame duration and a difference between a time stamp corresponding to the particular stage for the current frame and a time stamp corresponding to the particular stage for a previous frame

17 . The method of claim 15 , wherein the metric for a particular stage is calculated by multiplying a rate of the gain for the particular stage exceeding the second threshold value by a mean of the gain for the particular stage over a duration of the session.

18 . The method of claim 14 , wherein the content comprises a frame of video rendered in response to a feedback signal received from the client device, and wherein the feedback signal includes at least one signal selected from the group consisting of:
controller input; or sensor data from a head-mounted display.

19 . The method of claim 14 , wherein the information corresponding to each stage of the pipeline comprises a time stamp that indicates a time when the processing of the content by the stage is complete.

20 . A non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform steps comprising:
receiving, from a client device, metadata corresponding to content processed by a plurality of stages of a pipeline, wherein the metadata includes information corresponding to each stage of the plurality of stages of the pipeline; generating, for each stage of the plurality of stages of the pipeline, a metric calculated based on the metadata; and adjusting a parameter associated with at least one stage of the pipeline based on the metrics for the plurality of stages.

Assignees

• Nvidia Corp

Inventors

• Larson Alan
• Todur Bipin

Classifications

• H04N21/4781

  Games · CPC title

• H04N21/64738 Primary

  Monitoring network characteristics, e.g. bandwidth, congestion level (data switched network analysis H04L41/14; monitoring functioning in data switched networks H04L43/0817; flow control in packet networks H04L47/10) · CPC title

• H04N21/23614

  Multiplexing of additional data and video streams (arrangements for simultaneous broadcast of plural pieces of information H04H20/28) · CPC title

• H04N21/6581

  Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application · CPC title

• H04N21/8547

  involving timestamps for synchronizing content · CPC title