Generating composite video stream for display in VR

Patent metadata

Field	Value
Publication number	US-11516521-B2
Application number	US-201917262598-A
Country	US
Kind code	B2
Filing date	Jul 29, 2019
Priority date	Jul 30, 2018
Publication date	Nov 29, 2022
Grant date	Nov 29, 2022

Abstract

A processor system and computer-implemented method may be provided for generating a composite video stream which may combine a background video and a foreground video stream into one stream. For that purpose, a spatially segmented encoding of the background video may be obtained, for example in the form of a tiled stream. The foreground video stream may be received, for example, from a(nother) client device. The foreground video stream may be a real-time stream, e.g., when being used in real-time communication. The image data of the foreground video stream may be inserted into the background video by decoding select segments of the background video, inserting the foreground image data into the decoded background image data of these segments, and by encoding the resulting composite image data to obtain composite segments which, together with the non-processed segments of the background video, form a spatially segmented encoding of a composite video.

First claim

The invention claimed is:

1. A method of generating a composite video stream for being displayed by a client device, the method comprising another client device or a server performing:
obtaining a spatially segmented encoding of a background video, said segmented encoding comprising independently decodable background segments;
receiving at least one foreground video stream at least part of which is to be displayed in front of the background video;
determining a placement position of the foreground video stream relative to the background video;
identifying a subset of background segments which are partially or entirely occluded by the foreground video stream when the foreground video stream is displayed at the placement position;
generating the composite video stream for the client device by, for a respective output frame of the composite video stream and respective input frames of the background video and the foreground video stream:
selectively decoding the subset of background segments, thereby obtaining decoded background image data,
compositing image data of the foreground video stream into the decoded background image data, thereby obtaining composite image data, and
encoding the composite image data to obtain a set of independently decodable composite segments which, together with the background segments which are not in the subset of background segments, represent a spatially segmented encoding of the output frame of the composite video stream; and

streaming the composite video stream, or spatial segments of the composite video stream which are within a field of view of a user of the client device, to the client device.

2. The method according to claim 1 , wherein the client device is configured to display the composite video stream in a VR environment and to render the VR environment from a viewing position of a user, wherein the method further comprises:
determining the viewing position of the user, for example by receiving data indicative of the viewing position from the client device; and
identifying the subset of background segments which are partially or entirely occluded by the foreground video stream further based on the viewing position.

3. The method according to claim 1 , further comprising:
obtaining placement metadata associated with the spatially segmented encoding of the background video, wherein the placement metadata defines the placement position of the foreground video stream relative to the background video.

4. The method according to claim 1 , wherein the spatially segmented encoding of the background video is obtained as a bitstream, and wherein the method further comprises:
replacing the subset of background segments with the set of composite segments in the bitstream to obtain the spatially segmented encoding of the output frame of the composite video stream.

5. The method according to claim 1 , wherein the spatially segmented encoding of the background video contains a map projection of the background video, and wherein the method further comprises:
compositing the image data of the foreground video stream into the decoded background image data based on the map projection.

6. The method according to claim 5 , wherein compositing the image data of the foreground video stream into the decoded background image data comprises applying the map projection to the image data of the foreground video stream before or when said image data is composited into the decoded background image data.

7. The method according to claim 5 , wherein the map projection of the background video is an equirectangular projection.

8. The method according to claim 1 , wherein the client device is configured to display the composite video stream in a VR environment and to render the VR environment in the field of view of the user, wherein the method further comprises:
determining the field of view of the user in the VR environment; and
compositing the image data of the foreground video stream into the decoded background image data based on the field of view.

9. The method according to claim 8 , wherein compositing the image data of the foreground video stream into the decoded background image data based on the field of view comprises at least one of:
only compositing the image data of the foreground video stream if said image data is in the field of view of the user; and
compositing the image data of the foreground video stream at a reduced image quality if said image data is outside of the field of view of the user.

10. The method according to claim 1 , wherein at least one of the spatially segmented encoding of the background video and the spatially segmented encoding of the composite video stream is represented by a tile-based media stream.

11. The method according to claim 1 , further comprising:
receiving a real-time foreground video stream from each one of a set of client devices, each respective foreground video steam representing a real-time recording of a respective user of a respective client device;
for each one of the client devices, generating a composite video stream which comprises image data of the foreground video streams received from the other client devices composited into the background video at respective placement positions.

12. The method according to claim 1 , wherein the server is an edge node of a 5G or next generation telecommunication network.

13. A non-transitory computer-readable medium comprising a computer program, the computer program comprising instructions for causing a processor system to perform the method according to claim 1 .

14. A non-transitory computer-readable medium comprising placement metadata associated with a spatially segmented encoding of a background video, the placement metadata defining a placement position of at least one foreground video stream relative to the background video.

15. A processor system configured for generating a composite video stream for being displayed by a client device, the processor system comprising:
a network interface to a network;
an input interface for obtaining a spatially segmented encoding of a background video, said segmented encoding comprising independently decodable background segments;
a processor configured to:
via the network interface, receive at least one foreground video stream at least part of which is to be displayed in front of the background video;
determine a placement position of the foreground video stream relative to the background video;
identify a subset of background segments which are partially or entirely occluded by the foreground video stream when the foreground video stream is displayed at the placement position;
generate the composite video stream for the client device by, for a respective output frame of the composite video stream and respective input frames of the background video and the foreground video stream:
selectively decode the subset of background segments, thereby obtaining decoded background image data,
composite image data of the foreground video stream into the decoded background image data, thereby obtaining composite image data, and
encode the composite image data to obtain a set of independently decodable composite segments which, together with the background segments which are not in the subset of background segments, represent a spatially segmented encoding of the output frame of the composite video stream; and

via the network interface, stream the composite video stream, or select spatial segments of the composite video stream which are within a field of view of a user of the client device to the client device.

16. A system co

Assignees

• Koninklijke Kpn Nv
• TNO

Inventors

• Da Silva Pratas Gabriel Alexandre
• Van Deventer Mattijs Oskar
• Stokking Hans Maarten

Classifications

• H04N21/23412 Primary

  for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects · CPC title

• H04N21/236

  Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream {(multiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00)} · CPC title

• H04N13/178

  Metadata, e.g. disparity information · CPC title

• H04N21/23614 Primary

  Multiplexing of additional data and video streams (arrangements for simultaneous broadcast of plural pieces of information H04H20/28) · CPC title

• H04N21/85406

  involving a specific file format, e.g. MP4 format · CPC title