Low-latency 360 VR streaming system and method for estimating region of interest of user

Patent metadata

Field	Value
Publication number	US-12425558-B2
Application number	US-202418645918-A
Country	US
Kind code	B2
Filing date	Apr 25, 2024
Priority date	Oct 27, 2021
Publication date	Sep 23, 2025
Grant date	Sep 23, 2025

Abstract

A low-latency 360 virtual reality (VR) streaming method for estimating a region of interest (ROI) of a user is proposed. The method may include receiving device pose information from a user device, extracting an ROI of a user, based on the device pose information, and generating ROI tile information corresponding to the ROT. The method may also include generating ROI estimation tile information corresponding to a position at which the ROI is to be changed, and requesting, from a streaming server, a tile corresponding to the ROI tile information and the ROI estimation tile information to receive a high-quality tile stream and a low-quality tile stream of full video that includes a low-quality full video tile stream. The method may further include decoding and rendering the high-quality tile stream and the low-quality full video tile stream and providing the rendered video to the user device.

First claim

What is claimed is:

1. A low-latency 360 virtual reality (VR) streaming method for estimating a region of interest (ROI) of a user, the low-latency 360 VR streaming method comprising:
receiving device pose information from a 360 VR device that comprises a user device;
extracting an ROI of a user, based on the device pose information;
generating ROI tile information corresponding to the ROI;
generating ROI estimation tile information corresponding to a position at which the ROI is to be changed;
requesting, from a streaming server, a tile corresponding to the ROI tile information and the ROI estimation tile information to receive a high-quality tile stream and a low-quality tile stream of full video that comprises a low-quality full video tile stream;
decoding and rendering the high-quality tile stream and the low-quality full video tile stream; and
providing the rendered video to the user device.

2. The low-latency 360 VR streaming method of claim 1 , wherein generating the ROI tile information comprises:
calculating center tile information corresponding to a center of the extracted ROI; and
calculating tile group information corresponding to the ROI, based on the center tile information.

3. The low-latency 360 VR streaming method of claim 2 , wherein calculating the tile group information corresponding to the ROI based on the center tile information comprises:
calculating desired amount information about an adjacent tile by using size information about a tile and a resolution of the ROI; and
calculating a tile, corresponding to the desired amount information about the adjacent tile, as tile group information.

4. The low-latency 360 VR streaming method of claim 3 , wherein generating the ROI estimation tile information comprises:
calculating edge position information about the ROI, based on the tile group information; and
generating tiles adjacent to the ROI tile information as ROI estimation tile information when an amount of change in the ROI is greater than or equal to a certain distance per frame with respect to the edge position information.

5. The low-latency 360 VR streaming method of claim 1 , wherein receiving the high-quality tile stream and the low-quality full video tile stream receives the high-quality tile stream and the low-quality full video tile stream of a common media application format (CMAF) chunk unit.

6. The low-latency 360 VR streaming method of claim 5 , wherein receiving the high-quality tile stream and the low-quality full video tile stream allocates an index to each of a plurality of CMAF chunks corresponding to each tile and receives the CMAF chunks in the order of indexes of CMAF chunks alternately for each tile.

7. The low-latency 360 VR streaming method of claim 5 , wherein receiving the high-quality tile stream and the low-quality full video tile stream comprises:
receiving a low-quality full video tile stream from the streaming server;
receiving a high-quality tile stream that comprises an ROI high-quality tile stream corresponding to the ROI tile information; and
receiving a high-quality tile stream that comprises an ROI estimation high-quality tile stream corresponding to the ROI estimation tile information as the reception of the low-quality full video tile stream and the ROI high-quality tile stream is completed.

8. The low-latency 360 VR streaming method of claim 1 , wherein decoding and rendering the high-quality tile stream and the low-quality full video tile stream comprises:
rendering the decoded high-quality tile stream; and
rendering the low-quality full video tile stream in a non-rendering region as the rendering of the decoded high-quality tile stream is completed.

9. The low-latency 360 VR streaming method of claim 8 , wherein decoding and rendering the high-quality tile stream and the low-quality full video tile stream loads a texture used in the rendering by using a two-circular texture buffer.

10. The low-latency 360 VR streaming method of claim 8 , wherein rendering of the decoded high-quality tile stream checks a renderer buffer corresponding to the high-quality tile stream to perform mapping on a texture region corresponding to a tile, and
wherein rendering the low-quality full video tile stream in the non-rendering region scales up the low-quality full video tile stream by a size of a texture in a renderer buffer corresponding to the low-quality full video tile stream and additionally maps the low-quality full video tile stream to a texture region which is the non-rendering region.

11. A low-latency 360 virtual reality (VR) streaming system for estimating a region of interest (ROI) of a user, the low-latency 360 VR streaming system comprising:
a pose information tracking configured to receive device pose information from a 360 VR device that comprises a user device;
a stream reception processor configured to:
extract an ROI of a user, based on the device pose information,
generate ROI tile information corresponding to the ROI and ROI estimation tile information corresponding to a position at which the ROI is to be changed, and
request, from a streaming server, a tile corresponding to the ROI tile information and the ROI estimation tile information to receive a high-quality tile stream and a low-quality tile stream of full video that comprises a low-quality full video tile stream;

a multi-channel decoder configured to decode the high-quality tile stream and the low-quality full video tile stream; and
a simulcast rendering processor configured to perform rendering based on the decoded high-quality tile stream and the decoded low-quality full video tile stream to provide rendered video to the user device.

Assignees

• Korea Electronics Technology

Inventors

• Jang Jun Hwan
• Park Woo Chool
• Yang Jin Wook
• Yoon Sang Pil
• Choi Min Su
• Lee Jun Suk
• Song Su Ho
• Koo Bon Jae

Classifications

• G06V10/25 Primary

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• H04N13/117

  the virtual viewpoint locations being selected by the viewers or determined by viewer tracking · CPC title

• H04N21/4318

  by altering the content in the rendering process, e.g. blanking, blurring or masking an image region (image enhancement or restoration in general G06T5/00) · CPC title

• H04N21/4402

  involving reformatting operations of video signals for household redistribution, storage or real-time display {(details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764)} · CPC title

• H04N21/2343

  involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements {(video transcoding H04N19/40; media packet handling at the source H04L65/762)} · CPC title