Systems and methods for providing interactive streaming media

Patent metadata

Field	Value
Publication number	US-11068290-B2
Application number	US-202016818340-A
Country	US
Kind code	B2
Filing date	Mar 13, 2020
Priority date	Nov 14, 2016
Publication date	Jul 20, 2021
Grant date	Jul 20, 2021

Abstract

A client system presents, within an execution environment of an application, a third-party media stream distinct from the application, received from a remote host server via a network. The client system detects interaction events during presentation of the third-party media stream, and transmits descriptions of the detected interaction events to the remote host server. The application may be pre-cued prior to presentation, e.g., to minimize start-up time. In some implementations, a side-band message channel is established to facilitate communication between the client system and the remote host server.

First claim

What is claimed is:

1. A method comprising:
executing, by a client simulation system comprising a processor, a simulation of a client device, the simulation of the client device including a virtual application stack comprising a runtime layer and an operating framework;
executing, by the client simulation system, an application in the simulation of the client device, wherein the application generates (1) image data stored in the data storage, and (2) a request to access the runtime layer or the operating framework of the virtual application stack of the simulation of the client device;
transmitting, by the client simulation system, to a first client device via a network interface, a media stream generated using the image data stored in the data storage;
providing, by the client simulation system, responsive to the application generating the request to access the runtime layer or the operating framework, the request to the virtual application stack of the simulation of the client device;
intercept, using a messaging interface provided by the client simulation system to the simulation of the client device, the request to access the runtime layer or the operating framework of the virtual application stack;
transmitting, by the client simulation system, via the network interface, responsive to intercepting the request to access the runtime layer or the operating framework of the virtual application stack, the request-to a corresponding messaging interface forming part of a corresponding application stack of the first client device using the messaging interface, causing the first client device to:
access, using a corresponding operating framework or a corresponding runtime layer of the corresponding application stack, sensor information from sensors of the first client device via a hardware abstraction layer of the corresponding application stack;
generate a response to the request that includes the sensor information; and
transmit, using the corresponding messaging interface, the response to the messaging interface of the simulation of the client device;

receiving, by the client simulation system, using the messaging interface for the simulation of the client device, from the first client device, the response to the request; and
providing, by the client simulation system, to the application in the simulation of the client device, the sensor information included in the response received from the first client device.

2. The method of claim 1 , the method further comprising transmitting, by the client simulation system, the request to a controller application executed at the first client device, the controller application incorporating the corresponding messaging interface of the first client device, and receiving the response to the request from the controller application executed at the first client device.

3. The method of claim 2 , wherein the request, when received by the first client device, causes the controller application executed at the first client device to transmit device-specific data to the client simulation system.

4. The method of claim 3 , wherein the request from the application to the simulation of the client device intercepted by the messaging interface includes an access request for one of:
access to system configuration data on the client device;
access to an audio sensor on the client device;
access to an image sensor on the client device;
access to a biometric sensor on the client device;
access to a light emitting diode on the client device;
access to a parental control setting on the client device; or
access to a file system on the client device.

5. The method of claim 4 , wherein the request to access content the runtime layer or the operating framework of the virtual application stack further causes the client device to display an authorization screen, process an authorization, and, responsive to processing the authorization, provide an access granted indicator to the client simulation system.

6. The method of claim 4 , wherein the request, when transmitted using the messaging interface, includes a uniform resource locator (URL) configured to cause the controller application executed at the client device to transmit the device-specific data to the client simulation system.

7. The method of claim 2 , comprising:
receiving, from the first client device via the network, a client device characteristic descriptor;
selecting a client simulation for a generalized client device with a characteristic conforming to the client device characteristic descriptor; and
executing the simulation of the client device using the client simulation for the generalized client device.

8. A non-transitory computer-readable medium storing instructions that cause a processor executing the instructions to:
execute a simulation of a client device, the simulation of the client device including a virtual application stack comprising a runtime layer and an operating framework;
execute an application in the simulation of the client device, the application generating (1) image data stored in the data storage, and (2) a request to access the runtime layer or the operating framework of the virtual application stack of the simulation of the client device;
transmit, to a first client device via a network interface, a media stream generated using the image data stored in the data storage;
provide, responsive to the application generating the request to access the runtime layer or the operating framework, the request to the virtual application stack of the simulation of the client device;
intercept, using a messaging interface provided by the processor to the simulation of the client device, the request to access the runtime layer or the operating framework of the virtual application stack;
transmit, via the network interface, responsive to intercepting the request to access the runtime layer or the operating framework of the virtual application stack, the request to a corresponding messaging interface forming a part of a corresponding application stack of the first client device using the messaging interface, causing the first client device to:
access sensor information from sensors of the first client device via a hardware abstraction layer of the corresponding application stack;
generate a response to the request that includes the sensor information; and
transmit, using the corresponding messaging interface, the response to the messaging interface of the simulation of the client device;

receive, using the messaging interface for the simulation of the client device, from the first client device via the network interface, the response to the request; and
provide, to the application in the simulation of the client device, the sensor information included in the response received from the first client device.

9. The non-transitory computer-readable medium of claim 8 storing the instructions that cause the processor executing the instructions to transmit the request to a controller application executed at the first client device, the controller application incorporating the corresponding messaging interface of the first client device, and to receive the response to the request from the controller application executed at the first client device.

10. The non-transitory computer-readable medium of claim 9 , wherein the request, when received by the first client device, causes the controller application executed at the first client device to transmit device-specific data to the processor.

11. A system comprising:
a client simulation server comprising a memory, a network interface, and a processor,
the processor configured to:
execute a simulation of a client device, the simulation o

Assignees

• Google Llc

Inventors

• Toksoz Tuna
• Price Thomas

Classifications

• G06F9/485 Primary

  Task life-cycle, e.g. stopping, restarting, resuming execution (G06F9/4881 takes precedence) · CPC title

• G06F2009/45575

  Starting, stopping, suspending or resuming virtual machine instances · CPC title

• G06F9/45558

  Hypervisor-specific management and integration aspects · CPC title

• G06F9/452

  Remote windowing, e.g. X-Window System, desktop virtualisation (protocols for virtual reality H04L67/131) · CPC title

• G06F9/45504 Primary

  Abstract machines for programme code execution, e.g. Java virtual machine [JVM], interpreters, emulators · CPC title