Streaming content adjustment based on camera feedback

Patent metadata

Field	Value
Publication number	US-9686338-B1
Application number	US-201414523654-A
Country	US
Kind code	B1
Filing date	Oct 24, 2014
Priority date	Oct 24, 2014
Publication date	Jun 20, 2017
Grant date	Jun 20, 2017

Abstract

A streaming source sends streaming content to a display device. A camera captures images of the display device playing the streaming content and/or a user's interaction with the display device along with the displayed response to the user's interaction. The captured content is analyzed to determine, for example, image quality characteristics of the displayed content, a user experience associated with the displayed content or whether the displayed content matches an expected outcome associated with testing input. Information determined from the analysis of the captured content may be used to adjust characteristics of the streaming content, such as streaming parameters at the streaming source or configuration of the display device. A testing system may analyze the captured content to evaluate performance of the streaming system. Results of the testing may be sent to a data store and/or used to mitigate a poor user experience such as poor image quality.

First claim

What is claimed is:

1. A system, comprising:
a display device configured to:
receive a streaming content from a streaming source; and
display the received content; and

a camera configured to:
capture the displayed content; and
send the captured content to the display device;

wherein the display device is configured to:
send the captured content to the streaming source to be analyzed for determining one or more adjustments to the streaming content to produce adjusted streaming content, or
analyze the captured content to determine information for one or more configuration changes and send the information to the streaming source for determining one or more adjustments to the streaming content to produce adjusted streaming content; and

wherein the display device is further configured to receive the adjusted streaming content from the streaming source.

2. The system of claim 1 , wherein the display device is configured to perform the analysis of the captured content.

3. The system of claim 2 , wherein the display device comprises an adjustment engine that performs the determining of the one or more adjustments, and wherein the determined one or more adjustments are sent to the streaming source by the display device.

4. The system of claim 1 , wherein the camera is configured to perform the analysis of the captured content.

5. The system of claim 1 , wherein the display device is further configured to:
receive instructions from the streaming source, the instructions directing reconfiguration of the display device, wherein the instructions are based at least in part on the analysis of the captured content; and
execute the directed reconfiguration while displaying the received streaming content.

6. A method, comprising:
performing, by one or more computing devices:
sending streaming content to a display device;
receiving information associated with characteristics of a camera-captured version of the streaming content captured from a display on the display device wherein the camera-captured version is captured as the display device displays the streaming content, or receiving and analyzing the camera-captured version of the streaming content and determining the information; and
determining a mitigating response to the information.

7. The method of claim 6 , wherein said determining the mitigating response comprises:
determining one or more adjustments for the streaming source based on the information; and
instructing the streaming source to implement the one or more adjustments.

8. The method of claim 7 , further comprising:
sending, to the display device, adjusted streaming content that has been adjusted based at least in part on one of the determined one or more adjustments to the streamed content.

9. The method of claim 7 ,
wherein said determining the mitigating response further comprises determining that the one or more adjustments are for a reconfiguration of the display device; and
wherein the method further comprises:
sending, to the display device, instructions directing the reconfiguration of the display device based at least in part on the determined one or more adjustments.

10. The method of claim 6 , further comprising:
receiving, by a streaming source, the captured content such that the analyzing the camera-captured version of the streaming content and the determining the information associated with the characteristics of the camera-captured version of the streaming content is performed by the streaming source.

11. The method of claim 6 , wherein said analyzing the camera-captured version of the streaming content and determining the information comprises:
analyzing at least one of resolution, frame rate, frame loss, color attribute or audio attribute of the camera-captured version of the streaming content; and
determining, based on the analysis, one or more adjustments to one or more streaming parameters comprising at least one of frame rate, codec, resolution, video data rate, or audio data rate.

12. The method of claim 6 , further comprising:
instructing execution of a test, the test comprising testing input that causes one or more changes to the streaming content displayed by the display device;
wherein said analyzing the camera-captured version of the streaming content and determining the information comprises determining one or more testing outcomes in response to the test execution.

13. The method of claim 12 , wherein said determining one or more testing outcomes in response to the test execution comprises determining the one or more testing outcomes based upon analysis of one or more criteria comprising variation in video of the camera-captured version of the streaming content, lack of variation in the video of the camera-captured version of the streaming content, artifacts in the video of the camera-captured version of the streaming content, or a fault or latency in the camera-captured version of the streaming content.

14. The method of claim 13 ,
wherein said determining the mitigating response to the information comprises determining one or more adjustments to a streaming video system based on the one or more testing outcomes; and
the method further comprising causing the one or more adjustments to the streaming video system.

15. A non-transitory computer-readable medium storing program instructions that are executed by one or more processors to implement:
a content analysis module configured to:
receive information for a camera-captured version of streamed content that was captured from a display device, wherein the camera-captured version is captured as the display device displays the streamed content; and

an adjustment engine configured to:
determine one or more adjustments to the streamed content based on analysis of the received information for the camera-captured version of the streamed content; and
instruct reconfiguration of a streaming source for the streamed content in accordance with the one or more adjustments.

16. The non-transitory computer-readable medium of claim 15 ,
wherein the analysis of the received information associated with the camera-captured version of the streamed content comprises analysis of at least one of resolution, frame rate, frame loss, color attribute or audio attribute of the captured version of the streamed content; and
wherein the one or more adjustments comprise adjustments to at least one of frame rate, codec, resolution, video data rate, or audio data rate of the streaming content.

17. The non-transitory computer-readable medium of claim 15 , wherein the program instructions are further executed by the one or more processors to:
determine one or more adjustments to the display device based on analysis of the received information for the camera-captured version of the streamed content; and
send a message to the display device indicating the one or more adjustments to the display device.

18. The non-transitory computer-readable medium of claim 15 ,
wherein the streamed content comprises images for a virtual desktop session; and
wherein the program instructions are further executed by the one or more processors to:
receive an indication of input received at the display device, the input associated with the virtual desktop session;
alter the images for the virtual desktop session in accordance with the received input; and
send the altered images for the virtual desktop session as the streamed content such that the images displayed on the display device based on the streamed content are responsive to input to the display device.

Assignees

• Amazon Tech Inc

Inventors

• Farrell Eugene Michael
• Thomas Nathan Bartholomew
• Tellvik Erik Jonathon
• Suryanarayanan Deepak

Classifications

• H04L65/80 Primary

  Responding to QoS · CPC title

• H04L65/602 Primary

  Electricity · mapped topic

• H04L65/608

  Electricity · mapped topic

• H04L65/756

  adapting media to device capabilities · CPC title

• H04L65/612 Primary

  for unicast · CPC title