System and method for multiplexed video stream decoding in web browser

Patent metadata

Field	Value
Publication number	US-10855741-B2
Application number	US-201615229874-A
Country	US
Kind code	B2
Filing date	Aug 5, 2016
Priority date	Aug 6, 2015
Publication date	Dec 1, 2020
Grant date	Dec 1, 2020

Abstract

A system and method for real time streaming and decoding of multiple simultaneous streams within a web browser is disclosed. In a preferred embodiment, a stream manager of the web browser stores decoder context data for each of the streams. The stream manager then provides the frames in conjunction with the decoder context data for consumption by a decoder of the web browser, which decodes the frames, presents the frames for display, and updates the decoder context data with the frame data of the decoded frames. In another embodiment, the stream manager is integrated within the decoder.

First claim

What is claimed is:

1. A video streaming system, comprising:
one or more user devices, wherein each of the one or more user devices executes a web browser; and
a streaming server that provides multiple simultaneous streams of video frame data to the one or more user devices,
wherein:
a stream manager executing in the web browser of each of the one or more user devices that creates and stores a separate instance of decoder context data for each of the multiple simultaneous streams, buffers the frame data into completed frames, each of which is associated with one of the multiple simultaneous streams, and presents the completed frames and the decoder context data for decoding and display; and
a decoder executing in the web browser of each of the one or more user devices decodes each of the completed frames of the multiple simultaneous streams using the instance of the decoder context data corresponding to the stream with which the completed frame being decoded is associated, and the decoder updates the instance of the decoder context data used to decode the completed frame using the decoded frame;
wherein the decoder and the stream manager are implemented in javascript.

2. A video streaming system as claimed in claim 1 , wherein the streaming server is a network video recorder.

3. A video streaming system as claimed in claim 1 , wherein the stream manager is integrated within the decoder.

4. A video streaming system as claimed in claim 1 , wherein the stream manager presents the frames of the streams and the decoder context data for the streams to the decoder.

5. A video streaming system as claimed in claim 1 , wherein the decoder consumes the frames of the streams provided by the stream manager and multiplexes decoding of the frames of the streams using the decoder context data.

6. A video streaming system as claimed in claim 1 , wherein the web browsers include client applications, wherein the client applications receive the decoded frames of the streams from the decoder and display the decoded frames of the streams.

7. The video streaming system of claim 1 , wherein the stream manager includes a stream context repository pool and a frame buffer memory.

8. The video streaming system of claim 7 , wherein the stream context repository pool comprises a plurality of slots, each slot being associated with one of the multiple simultaneous streams and storing the instance of the decoder context data corresponding to the stream with which the slot is associated.

9. The video streaming system of claim 8 , wherein the frame buffer memory comprises frame buffers, each of which is associated with one of the multiple simultaneous streams and stores frame data received for the stream with which the frame buffer is associated.

10. The video streaming system of claim 9 , wherein, in response to receiving a data packet containing stream identification information indicating one of the multiple simultaneous streams and frame data for the stream indicated by the stream identification information and having an EOF flag set indicating that the data packet is a last data packet of a completed frame, the stream manager updates contents of the frame buffer associated with the stream indicated by the stream identification information and flushes the contents of the frame buffer into a completed frame for the stream indicated by the stream identification information.

11. The video streaming system of claim 10 , wherein the stream manager creates a separate frame entry for each completed frame, each frame entry including the completed frame, the stream identification information indicating the stream of the completed frame, and slot identification information indicating the slot of the stream context repository pool associated with the stream of the completed frame.

12. The video streaming system of claim 11 , wherein the decoder consumes each of the frame entries and, for each consumed frame entry, decodes the completed frame included in the frame entry using the instance of the decoder context data stored by the slot indicated by the slot identification information included in the frame entry.

13. The video streaming system of claim 12 , wherein, for each frame entry consumed by the decoder, upon decoding the completed frame included in the frame entry, the decoder updates the instance of the decoder context data stored by the slot indicated by the slot identification information included in the frame entry by generating new decoder context data from frame data of the decoded frame and saves the new decoder context data to the slot indicated by the slot identification information included in the frame entry.

14. The video streaming system of claim 1 , wherein the stream manager supports simultaneous decoding and display of at least four to nine streams.

15. The video streaming system of claim 1 , wherein, for each of the one or more user devices, a client application of the web browser executing on the user device presents a display area divided into multiple panes, each of the multiple panes being associated with a different one of the multiple simultaneous streams, the decoder provides the decoded frames from the multiple simultaneous streams to the client application, and the client application renders each of the decoded frames in the pane associated with the stream of the decoded frame.

16. The video streaming system of claim 1 , wherein the decoder is an h.264 decoder implemented in javascript.

17. The video streaming system of claim 1 , wherein each instance of the decoder context data includes a reference frame for decoding subsequent frames of the stream, and wherein, upon decoding a current frame of the stream, the decoder includes identified changes with respect to the reference frame in the current frame and declares the current frame as a new reference frame.

18. A method for decoding one or more streams of video frame data within a web browser executing on a user device, the method comprising;
a streaming server providing multiple simultaneous streams of video frame data to the user device;
a stream manager executing in the web browser and implemented in javascript creating and storing a separate instance of decoder context data for each of the multiple simultaneous streams and buffering the frame data into completed frames, each of which is associated with one of the multiple simultaneous streams;
the stream manager presenting the completed frames and the decoder context data for decoding and display; and
a decoder executing in the web browser and implemented in javascript decoding each of the completed frames of the multiple simultaneous streams using the instance of the decoder context data corresponding to the stream with which the completed frame being decoded is associated and updating the instance of the decoder context data used to decode the completed frame using the decoded frame.

19. The method of claim 18 , further comprising the web browser displaying the decoded frames of the one or more multiple simultaneous streams.

20. The method of claim 18 , wherein the stream manager presenting the completed frames and the decoder context data for the multiple simultaneous streams for decoding and display comprises:
placing each of the completed frames of the multiple simultaneous streams in a queue; and
for each completed frame, providing identification information indicating the separate instance of the decoder context data corresponding to the stream with which the completed frame is associated.

21. Th

Assignees

• Sensormatic Electronics Llc

Inventors

• Trani James
• Furtado Andrew
• Seeley Teresa J

Classifications

• H04L65/70

  Media network packetisation · CPC title

• H04L65/612

  for unicast · CPC title

• Y02D30/50

  in wire-line communication networks, e.g. low power modes or reduced link rate · CPC title

• H04L69/14

  Multichannel or multilink protocols · CPC title

• H04L67/02 Primary

  based on web technology, e.g. hypertext transfer protocol [HTTP] · CPC title