Enhancing a region of interest in video frames of a video stream

Patent metadata

Field	Value
Publication number	US-10674185-B2
Application number	US-201615765193-A
Country	US
Kind code	B2
Filing date	Oct 7, 2016
Priority date	Oct 8, 2015
Publication date	Jun 2, 2020
Grant date	Jun 2, 2020

Abstract

A method for enhancing a region of interest in video frames of a video stream is described. The method includes receiving media data defining base video frames of at least one base stream, said base stream being associated with one or more enhancement tile streams, an enhancement tile stream comprising media data defining tiled video frames comprising a tile, said tile comprising media data for enhancing visual content in a subregion of the image region of the base video frames; requesting media data of one or more enhancement tile streams, preferably one or more HEVC enhancement tile streams, the one or more enhancement tile streams being associated with media data for enhancing visual content in the at least one region of interest; generating base video frames on the basis of media data of the at least one base stream and generating tiled video frames on the basis of media data of the one or more enhancement tile streams, the tiled video frames comprising one or more tiles forming visual content the at least one region of interest; and, replacing or blending at least part of the visual content of said at least one region of interest in the video frames of said base stream with at least part of said enhanced visual content of the tiled video frames.

First claim

The invention claimed is:

1. Method for enhancing at least one region of interest in video frames of a video stream, the method comprising:
receiving, by a client device, media data defining base video frames of at least one base stream, said base stream being associated with one or more enhancement tile streams, wherein an enhancement tile stream comprises media data defining tiled video frames, a tiled video frame comprising a tile associated with media data for enhancing visual content in a subregion of an image region of a base video frame, wherein media data of an enhancement tile stream are independently decodable from the media data of said base stream;
requesting, by the client device, media data of one or more enhancement tile streams, the one or more tiles of the tiled video frames of the requested media data being associated with at least one predetermined region of interest in the image region of the base video frames;
generating, by a first decoder connected to the client device, base video frames on the basis of the media data of the at least one base stream and a second decoder connected to the client device generating tiled video frames on the basis of the media data of the one or more enhancement tile streams, the generated tiled video frames comprising one or more tiles forming enhanced visual content in the at least one region of interest; and,
forming, by a rendering device, video frames comprising an enhanced region of interest by modifying at least part of the visual content of the at least one region of interest in the video frames of said base stream on the basis of at least part of the enhanced visual content of the tiled video frames, said modifying comprising replacing or blending at least part of the visual content of the at least one region of interest in the video frames of said base stream with at least part of the enhanced visual content of the tiled video frames;
wherein said media data are requested by the client device on the basis of a manifest file comprising at least one base stream identifier and one or more enhancement tile stream identifiers associated with the at least one base stream identifier; the manifest file further comprising tile position information for signaling the client device on the spatial relation between visual content in the video frames of said base stream and visual content in the tiles of the tiled video frames of said one or more enhancement tile streams.

2. Method according to claim 1 wherein the position information comprises one or more spatial relationship descriptors (SRDs), a spatial relationship descriptor being associated with said base stream and/or an enhancement tile stream, the spatial relationship descriptor signaling the client device about the position of the visual content in the video frames of the base stream and/or about the position of a tile in the tiled video frames of the tiled enhancement tile stream.

3. Method according to claim 1 wherein the manifest file further comprises enhancement information associated with at least one of the one or more enhancement tile streams, the enhancement information being configured for signaling the client device about the type of enhancement provided by an enhancement tile stream, and, optionally, for providing the client device one or more enhancement parameters associated with at least one of the resolution, bitrate, color depth, frame rate of the one or more enhancement tile streams defined by said enhancement tile stream identifiers.

4. Method according to claim 1 comprising:
determining ROI position information associated with the position of the at least one region of interest in the video frames of said base stream;
determining one or more enhancement tile streams on the basis of the ROI position information and the tile position information.

5. Method according to claim 1 further comprising:
determining pixel replacement information for signaling the rendering device about pixels in the video frames of the base stream which are associated with the least one region of interest in the video frames of the base stream;
the rendering device using the pixel replacement information for replacing or blending pixel information in at least part of the at least one region of interest in the video frames of the base stream with associated pixel information of the enhanced visual content of the tiled video frames.

6. Method according to claim 5 wherein the pixel replacement information is at least partly determined on the basis of tile position information in a manifest file.

7. Method according to claim 1 wherein said forming comprises:
prior to said modifying, adjusting said base video frames on the basis of information on the video and/or picture characteristics of the tiled video frames of the one or more enhancement tile streams.

8. Method according to claim 1 wherein the media data associated with a tile in a first tiled video frame are encoded such that the media data do not have any spatial decoding dependency on the media data of one or more other tiles in the first tiled video frame; and/or, wherein the media data associated with the tile are encoded such that the media data do not have any temporal decoding dependency on media data of other tiles in one or more tiled video frames preceding the first tiled video frame or following the first tiled video frames.

9. Method according to claim 1 wherein an enhancement tile stream comprise media data with enhanced luminance information.

10. Method according to claim 1 wherein modifying at least part of the visual content of said at least one region of interest comprises:
copying the content of a first video frame buffer comprising a video frame of the base stream into an output frame buffer of the media device;
overwriting or blending pixel values associated with the at least one region of interest in the output frame buffer with pixel values at corresponding pixel positions in a second video frame buffer comprising a tiled video frame associated with one or more enhancement tile streams, wherein a binary mask indicates which pixels values in the second video frame buffer needs to be copied into the output frame buffer.

11. Method according to claim 1
wherein said one or more enhancement tile streams are formatted on the basis of a data container of a media streaming protocol or media transport protocol, an (HTTP) adaptive streaming protocol or a transport protocol for packetized media data; and/or,
wherein media data of the one or more enhancement tile streams are encoded on the basis of a codec supporting tiled video frames; and/or,
wherein media data of the one or more enhancement tile streams defined by the one or more enhancement tile stream identifiers are stored in (tile) tracks and wherein metadata associated with at least part of the one or more enhancement tile streams are stored in at least one base track.

12. A computer comprising:
a non-transitory computer readable storage medium having computer readable program code embodied therewith, and a processor coupled to the non-transitory computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform executable operations comprising:
receiving media data defining base video frames of at least one base stream, said base stream being associated with one or more enhancement tile streams, wherein an enhancement tile stream comprises media data defining tiled video frames, a tiled video frame comprising a tile associated with media data for enhancing visual content in a subregion of an image region of a base video frame, wherein media data of an enhancement tile s

Assignees

• Koninklijke Kpn Nv

Inventors

• Thomas Emmanuel
• Van Brandenburg Ray
• Van Deventer Mattijs Oskar

Classifications

• H04N21/234327 Primary

  by decomposing into layers, e.g. base layer and one or more enhancement layers · CPC title

• H04N21/85406 Primary

  involving a specific file format, e.g. MP4 format · CPC title

• H04N19/167

  Position within a video image, e.g. region of interest [ROI] · CPC title

• H04N19/187

  the unit being a scalable video layer · CPC title

• H04N19/30

  using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence) · CPC title