Measuring user quality of experience for a streaming media service

Patent metadata

Field	Value
Publication number	US-10200436-B2
Application number	US-201615333125-A
Country	US
Kind code	B2
Filing date	Oct 24, 2016
Priority date	Dec 16, 2011
Publication date	Feb 5, 2019
Grant date	Feb 5, 2019

Abstract

Techniques are disclosed for representing a user quality of experience (QoE) experienced by users of a streaming media service using a single QoE metric. The single QoE metric may be determined based on a set of empirical characteristics relating to the streaming video service such as startup latency, video quality, and the likelihood of interruptions in streaming playback. The empirical characteristics may be weighted according to how much one factor influences user quality of experience, relative to the others. Representing the QoE as a single metric may allow a streaming media service provider to improve key business measures such as subscriber retention and engagement.

First claim

We claim:

1. A computer-implemented method, comprising:
sampling a plurality of performance characteristics associated with a streaming video service while streaming media content to one or more users;
for each performance characteristic included in the plurality of performance characteristics, determining a numerical contribution that the performance characteristic makes to a single quality of experience (QoE) metric for the streaming video service;
determining, for each performance characteristic included in the plurality of performance characteristics, a weight corresponding to the performance characteristic;
computing the single QoE metric by applying, for each performance characteristic included in the plurality of performance characteristics, the weight corresponding to the performance characteristic to the numerical contribution determined for the performance characteristic, wherein each weight is determined to increase a correlation between the single QoE metric and a measure of a level with which the one or more users are engaged with the streaming video service; and
updating at least one configuration setting for the streaming video service based on the computed single QoE metric.

2. The computer-implemented method of claim 1 , wherein the measure of the level with which the one or more users are engaged with the streaming video service comprises either a measure of hours watched by users of the streaming video service or a level of subscriber retention for the streaming video service.

3. The computer-implemented method of claim 1 , wherein at least one of the performance characteristics comprises streaming video startup latency.

4. The computer-implemented method of claim 1 , wherein at least one of the performance characteristics comprises initial video quality.

5. The computer-implemented method of claim 1 , wherein at least one of the performance characteristics comprises streaming video quality.

6. The computer-implemented method of claim 5 , wherein streaming video quality is related to a bit rate at which streaming video is delivered to a user.

7. The computer-implemented method of claim 1 , wherein at least one of the performance characteristics comprises at least one of a probability of a buffer under-run occurring, a time required to reach a specified video quality, a duration of rebuffer events, and a distribution of rebuffer events.

8. The computer-implemented method of claim 1 , wherein at least one of the performance characteristics relates to how frequently a bit rate at which streaming video is delivered to a client device changes.

9. The computer-implemented method of claim 1 , wherein a contribution function correlates different values of the performance characteristic with different numerical contributions to the single QoE metric.

10. The computer-implemented method of claim 1 , further comprising:
monitoring the computed single QoE metric while streaming video content to a user; and
upon determining the computed single QoE metric has fallen below a specified threshold, performing one or more corrective actions to improve the computed single QoE metric.

11. The computer-implemented method of claim 1 , wherein each weight comprises a value that increases a correlation between the computed single QoE metric and the measure of the level with which the one or more users are engaged with the streaming video service, wherein the computed single QoE metric and the measure of the level with which the one or more users are engaged with the streaming video service are correlated when a high level of the computed single QoE metric corresponds to a high level of the measure of the level with which the one or more users are engaged with the streaming video service.

12. The computer-implemented method of claim 1 , further comprising setting one or more configuration settings for the streaming video service for a first connection associated with at least one of a user of a particular class of users, a particular streaming media client device, and a device of a particular class of streaming media client devices, wherein the one or more configuration settings increase a value of the computed single QoE metric for connections that correspond to the first connection.

13. A non-transitory computer-readable storage medium including instructions that, when executed by a processor, cause the processor to perform the steps of:
sampling a plurality of performance characteristics associated with a streaming video service while streaming media content to one or more users;
determining, for each performance characteristic included in the plurality of performance characteristics, a weight corresponding to the performance characteristic;
computing a single quality of experience (QoE) metric by applying, for each performance characteristic included in the plurality of performance characteristics, the weight corresponding to the performance characteristic to a numerical contribution determined for the performance characteristic, wherein each weight is determined to increase a correlation between the single QoE metric and a measure of a level with which the one or more users are engaged with the streaming video service; and
updating at least one configuration setting for the streaming video service based on the computed single QoE metric.

14. The non-transitory computer-readable storage medium of claim 13 , wherein the measure of the level with which the one or more users are engaged with the streaming video service comprises either a measure of hours watched by users of the streaming video service or a level of subscriber retention for the streaming video service.

15. The non-transitory computer-readable storage medium of claim 13 , wherein at least one of the performance characteristics comprises at least one of streaming video startup latency, initial video quality, average video quality, a time required to reach a specified video quality, a probability of a buffer under-run occurring, how frequently the bit rate at which streaming video is delivered to a client device changes, a duration of rebuffer events, and a distribution of rebuffer events.

16. The non-transitory computer-readable storage medium of claim 13 , further comprising, for each performance characteristic included in the plurality of performance characteristics, determining the numerical contribution that the performance characteristic makes to the single QoE metric for the streaming video service, and wherein computing the single QoE metric comprises applying each weight to the numerical contribution that the corresponding performance characteristic makes to the single QoE metric.

17. The non-transitory computer-readable storage medium of claim 13 , wherein a contribution function correlates different values of the performance characteristic with different numerical contributions to the single QoE metric.

18. The computer-readable storage medium of claim 13 , wherein the computed single QoE metric is determined for a plurality of users assigned to a common user type, for a particular streaming media client device, or for a particular class of streaming media client devices.

19. A system, comprising:
a memory that includes an application;
a processor couple to the memory and, when executing the application, is configured to:
sample a plurality of performance characteristics associated with a streaming video service while streaming media content to one or more users;
determine, for each performance characteristic included in the plurality of performance characteristics, a

Assignees

• Netflix Inc

Inventors

• Funge John
• Watson Mark
• Wei Wei
• Chen David

Classifications

• H04L65/80 Primary

  Responding to QoS · CPC title

• H04L67/322

  Electricity · mapped topic

• H04L43/045

  for graphical visualisation of monitoring data · CPC title

• H04N21/44218

  Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45) · CPC title

• H04L67/61

  taking into account QoS or priority requirements · CPC title