Real-time immersive content providing system, and haptic effect transmission method thereof

Patent metadata

Field	Value
Publication number	US-12050728-B2
Application number	US-201917595491-A
Country	US
Kind code	B2
Filing date	Sep 18, 2019
Priority date	May 17, 2019
Publication date	Jul 30, 2024
Grant date	Jul 30, 2024

Abstract

A haptic effect transmission method for providing real-time immersive content according to the present invention includes executing participatory content in which one or more users participate, collecting motion information of the users participating in the participatory content through a haptic device, multiplexing the motion information with video and audio files of the participatory content to obtain a multiplexed file, and demultiplexing the multiplexed file and providing the demultiplexed file to a display device and a haptic device of a client terminal.

First claim

The invention claimed is:

1. A haptic effect transmission method for providing real-time immersive game content, the haptic effect transmission method comprising:
executing real-time immersive participatory game content in which a plurality of game player users participate through a plurality of user terminals and a plurality of viewers participate through a plurality of client terminals, wherein the plurality of user terminals communicate data with the plurality of client terminals;
collecting motion information of each of the plurality of game player users participating in the real-time immersive participatory game content through a haptic device of a corresponding one of the plurality of user terminals, the motion information comprising a haptic response of the haptic device worn by a corresponding game player user;
multiplexing a haptic file containing the motion information with video and audio files of the real-time immersive participatory game content to obtain a multiplexed file, wherein the multiplexing comprises synchronizing the video and audio files and the motion information of the haptic file based on presentation time stamps and decoding time stamps of the haptic file and the video and audio files; and
in response to selection of a first game player user of the plurality of game player users from a first client terminal of the plurality of client terminals, sending the multiplexed file of the first game player user participating the real-time immersive participatory game content to the first client terminal to enable the first client terminal to perform demultiplexing the multiplexed file of the first game player user and simultaneously provide demultiplexed video and audio to a display device of the first client terminal and demultiplexed motion information of the first game player user to a haptic device of the first client terminal in real time such that the video and audio files and the haptic response are simultaneously provided to the first client terminal,
wherein the demultiplexing comprises:
synchronizing the demultiplexed video, audio and motion information based on the presentation time stamps and the decoding time stamps, and
outputting the synchronized video and audio to the display device and the synchronized motion information to the haptic device of the first client terminal worn by a first viewer of the plurality of viewers.

2. The haptic effect transmission method of claim 1 , wherein the motion information is generated based on movements, speeds, shaking, action intensity, and emotional expressions of objects and people of the real-time immersive participatory game content.

3. A real-time immersive content provision system for transmitting a haptic effect, the real-time immersive content provision system comprising:
a plurality of user terminals each including a display device on which real-time immersive participatory game content is executed and a haptic device which is configured to collect motion information of a corresponding one of a plurality of game player users participating in the real-time immersive participatory game content, the plurality of user terminals configured to communicate data with a plurality of client terminals of a plurality of viewers participating in the real-time immersive participatory game content, the motion information comprising a haptic response of the haptic device worn by a corresponding game player user; and
a server configured to multiplex a haptic file containing the motion information of each game player user with video and audio files of the real-time immersive participatory game content received from the plurality of user terminals to obtain a multiplexed file, the server being further configured to synchronize the video and audio files and the motion information of the haptic file based on presentation time stamps and decoding time stamps of the haptic file and the video and audio files to obtain the multiplexed file,
wherein in response to selection of a first game player user of the plurality of game player users from a first client terminal of the plurality of client terminals, the server is configured to send the multiplexed file of the first game player user participating the real-time immersive participatory game content to the first client terminal to enable the first client terminal to demultiplex the multiplexed file of the first game player user and simultaneously provide demultiplexed video and audio to a display device of the first client terminal and demultiplexed motion information of the first game player user to a haptic device of the first client terminal in real time such that the video and audio files and the haptic response are simultaneously provided to the first client terminal, and
wherein the first client terminal is configured to:
synchronize the demultiplexed video, audio and motion information based on the presentation time stamps and the decoding time stamps, and
output the synchronized video and audio to the display device and the synchronized motion information to the haptic device of the first client terminal worn by a first viewer of the plurality of viewers.

4. The real-time immersive content provision system of claim 3 , wherein the motion information is configured to be generated based on movements, speeds, shaking, action intensity, and emotional expressions of objects and people of the real-time immersive participatory game content.

Assignees

• Korea Electronics Technology

Inventors

• Park Woo Chool
• Jang Jun Hwan
• Kim Yong Hwa
• Yang Jin Wook
• Yoon Sang Pil
• Kim Hyun Wook
• Cho Eun Kyung
• Choi Min Su
• Lee Jun Suk
• Yang Jae Young

Classifications

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• A63F13/285

  Generating tactile feedback signals via the game input device, e.g. force feedback · CPC title

• H04N21/2368

  Multiplexing of audio and video streams · CPC title

• H04N21/23614

  Multiplexing of additional data and video streams (arrangements for simultaneous broadcast of plural pieces of information H04H20/28) · CPC title