Synchronized rendering of split multimedia content on network clients

Patent metadata

Field	Value
Publication number	US-2017006331-A1
Application number	US-201514788289-A
Country	US
Kind code	A1
Filing date	Jun 30, 2015
Priority date	Jun 30, 2015
Publication date	Jan 5, 2017
Grant date	—

Abstract

A system is disclosed for rendering a split multimedia content stream associated with a single program among networked playback devices in sync with each other. Splitting multimedia content allows, for example, two viewers of the same movie to hear the audio track in different languages, or presentation of related program information on a second screen. A presentation method disclosed ensures that the same program can be played in full or in part on multiple devices while maintaining audio and video synchronization among the devices. In one embodiment, synchronization is achieved by monitoring network latency and client system latency, and then incorporating latency information into a program clock reference (PCR) signal for transmission to a secondary playback device.

First claim

1 . A multimedia system, comprising:
a media server communicatively coupled to a network; a primary media player communicatively coupled to the media server via the network; a secondary media player communicatively coupled to the media server via the network; and a memory storing instructions that, when executed by a microprocessor, cause the media server to:
split a multimedia signal associated with a program into separate video and audio streams;
transmit a first video stream associated with the program and a first audio stream associated with the program to the primary media player;
transmit a second audio stream associated with the program to the secondary media player;
coordinate substantially simultaneous output of the first and second audio streams by the primary and secondary media players, respectively, the first and second audio streams being synchronized with the first video stream.

2 . The system of claim 1 wherein at least one of the primary and secondary media players includes a television, a game console, a DVR, a computer, a laptop, a tablet, or a smart phone.

3 . The system of claim 1 wherein the media server is an enhanced broadcast media set top box.

4 . The system of claim 1 wherein the secondary media player includes a display, and the instructions further cause the media server to:
transmit a second video stream associated with the program to the secondary media player, and
synchronize rendering of the first video stream on the secondary media player display with a program clock signal.

5 . The system of claim 1 wherein the second audio stream is a translation of the first audio stream into a different language.

6 . The system of claim 4 wherein the second video stream is augmented with one or more of explanatory text, subtitles, annotations, graphics, and superimposed images.

7 . The system of claim 1 wherein the media server includes a primary system configured to perform latency adjustments

8 . The system of claim 7 wherein the media server further includes an extended system including a program clock reference generator and a transport stream multiplexer.

9 . (canceled)

10 . (canceled)

11 . A method of streaming multimedia content from a media server to different media players, the method comprising:
receiving, via a network, multimedia signals associated with a single multimedia program; splitting the multimedia signals into separate video and audio streams; transmitting a first video stream and a first audio stream to a primary media player having a first display and a first speaker; transmitting a second audio stream to a secondary media player having a second display and a second speaker; and coordinating substantially simultaneous display of the first and second audio streams associated with the single multimedia program on the primary and secondary media players using a digital program clock as a common reference.

12 . The method of claim 11 , further comprising:
transmitting a second video stream associated with the single multimedia program to the secondary media player; and synchronizing rendering of the second video stream on the secondary media player to a rendering of the first video stream on the primary media player using a digital program clock as a common reference.

13 . The method of claim 11 wherein rendering the second audio stream by the secondary media player provides, via the second speaker, a soundtrack of the program in a language different from the first audio stream when rendered on the primary media player.

14 . The method of claim 11 wherein the primary media player is a television, the secondary media player is a smart phone, and the second speaker is a headset.

15 . The method of claim 12 wherein display of the second video stream by the second media player provides enhanced video content relating to the same program displayed by the primary media player.

16 . The method of claim 12 wherein the splitting includes demodulating and encoding the media signals.

17 . The method of claim 16 wherein the decoding includes transcoding to provide video streamed at different resolutions.

18 . (canceled)

19 . (canceled)

20 . The system of claim 1 , wherein coordinating substantially simultaneous output of the first and second audio streams includes:
determining, by the media server, a network latency for the primary and secondary media players; determining a client system latency for the primary and secondary media players based on respective presentation time stamps associated with rendering video frames on the primary and secondary media players; adjusting respective program clock reference values for the primary and secondary media players to account for the respective network latencies and the respective client system latencies; and transmitting the adjusted program clock reference values to the primary and secondary media players to synchronize data presentation.

21 . The method of claim 11 wherein coordinating substantially simultaneous display of the first and second audio streams includes:
determining a network latency for the primary and secondary media players;
determining a client system latency for the primary and secondary media players based on respective presentation time stamps associated with rendering video frames on the primary and secondary media players;
adjusting reference values of the digital program clock for the primary and secondary media players to account for the respective network latencies and the respective client system latencies; and
transmitting the adjusted reference values of the digital program clock to the primary and secondary media players to synchronize data presentation.

22 . A non-transitory computer-readable storage medium containing instructions which, when executed by a processor of a media server, cause the media server to:
receive, via a network, multimedia signals associated with a single multimedia program; split the multimedia signals into separate video and audio streams; transmit a first video stream and a first audio stream to a primary media player having a first display and a first speaker; transmit a second audio stream to a secondary media player having a second display and a second speaker; and coordinate substantially simultaneous display of the first and second audio streams associated with the single multimedia program on the primary and secondary media players using a digital program clock as a common reference.

23 . The non-transitory computer-readable storage medium of claim 22 , the instructions, when executed by the processor, further causing the media server to:
determine a network latency for the primary and secondary media players; determine a client system latency for the primary and secondary media players based on respective presentation time stamps associated with rendering video frames on the primary and secondary media players; adjust reference values of the digital program clock for the primary and secondary media players to account for the respective network latencies and the respective client system latencies; and transmit the adjusted reference values of the digital program clock to the primary and secondary media players to synchronize data presentation

24 . The non-transitory computer-readable storage medium of claim 22 , the instructions, when executed by the processor, further causing the media server to:
determine the

Assignees

• St Microelectronics Int Nv

Inventors

• Jairath Gaurav
• SRIVASTAVA Amit-Kumar
• Pandey Deepak

Classifications

• H04N21/439

  Processing of audio elementary streams · CPC title

• H04N21/44227

  Monitoring of local network, e.g. connection or bandwidth variations; Detecting new devices in the local network (configuring of peripheral devices in general G06F9/4411; monitoring connectivity in data switched networks H04L43/0811) · CPC title

• H04N21/44

  Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs · CPC title

• H04N21/8106

  involving special audio data, e.g. different tracks for different languages · CPC title

• H04L7/0037

  Delay of clock signal · CPC title