Mixed reality coding with overlays

Patent metadata

Field	Value
Publication number	US-10872441-B2
Application number	US-201916259473-A
Country	US
Kind code	B2
Filing date	Jan 28, 2019
Priority date	Apr 24, 2017
Publication date	Dec 22, 2020
Grant date	Dec 22, 2020

Abstract

A system includes a camera to capture real world content and a semiconductor package apparatus. The semiconductor package apparatus includes a substrate and logic. The logic includes a graphics pipeline to generate rendered content, a base layer encoder to encode real world content into a base layer and a first layer encoder to encode rendered content into a first non-base layer, a multiplexer to interleave the base layer with the first non-base layer to obtain a single output signal having mixed reality content, and a transmitter to transmit the single output signal. The system further includes a second layer encoder to encode map data into a second non-base layer. The multiplexer to interleave the second non-base layer with the first non-base layer and the base layer. The first and second layer encoders encode the rendered content and the map data into overlay auxiliary pictures.

First claim

What is claimed is:

1. An apparatus comprising:
a substrate; and
logic coupled to the substrate, wherein the logic is implemented in one or more of configurable logic or fixed-functionality hardware logic, the logic including:
a demultiplexer to receive a single complex signal having multiple interleaved signals and to separate one or more of a base layer signal, a first non-base layer signal, a second non-base layer signal, or a third non-base layer signal from the single complex signal;
a base layer decoder to decode the base layer signal into real world content and a first layer decoder to decode the first non-base layer signal into a mixed reality content overlay;
a second layer decoder to decode the second non-base layer signal into a map overlay and a third layer decoder to decode the third non-base layer signal into a blend overlay; and
a compositor to composite selected overlays with the decoded base layer signal for display.

2. The apparatus of claim 1 , wherein the real world content includes captured video.

3. The apparatus of claim 1 , wherein the map overlay distinguishes between placement of objects within an overlay auxiliary picture on another layer.

4. The apparatus of claim 1 , wherein the blend overlay represents transparency of an overlay object from one or more overlay auxiliary pictures.

5. At least one non-transitory computer readable storage medium comprising a set of instructions which, when executed, cause a computing apparatus to:
receive a single complex signal comprising mixed reality content;
separate the received signal into a plurality of signals comprising a base layer signal, a first non-base layer signal, a second non-base layer signal, and a third non-base layer signal;
decode the base layer signal into real world content and decoding the first non-base layer signal into a mixed reality content overlay;
decode the second non-base layer signal into a map overlay and decoding the third non-base layer signal into a blend overlay; and
composite one or more selected overlays with the decoded base layer signal for display.

6. The at least one non-transitory computer readable storage medium of claim 5 , wherein the real world content includes captured video.

7. The at least one non-transitory computer readable storage medium of claim 5 , wherein the map overlay distinguishes between placement of obj ects within an overlay auxiliary picture on another layer.

8. The at least one non-transitory computer readable storage medium of claim 5 , wherein the blend overlay represents transparency of an overlay object from one or more overlay auxiliary pictures.

9. A method comprising:
receiving a single complex signal comprising mixed reality content;
separating the received signal into a plurality of signals comprising a base layer signal, a first non-base layer signal, a second non-base layer signal, and a third non-base layer signal;
decoding the base layer signal into real world content and decoding the first non-base layer signal into a mixed reality content overlay;
decoding the second non-base layer signal into a map overlay and decoding the third non-base layer signal into a blend overlay; and
compositing one or more selected overlays with the decoded base layer signal for display.

10. The method of claim 9 , wherein the real world content includes captured video.

11. The method of claim 9 , wherein the map overlay distinguishes between placement of objects within an overlay auxiliary picture on another layer.

12. The method of claim 9 , wherein the blend overlay represents transparency of an overlay object from one or more overlay auxiliary pictures.

13. A system comprising:
a first semiconductor apparatus including a first substrate and first logic coupled to the first substrate, wherein the first logic includes:
a graphics pipeline to generate nixed reality content,
a base layer encoder to encode real world content into a base layer and a first layer encoder to encode the mixed reality content into a first non-base layer,
a second layer encoder to encode map data into a second non-base layer and a third layer encoder to encode blending data into a third non-base layer, and
a multiplexer to interleave the third non-base layer with the second non-base layer, the first non-base layer and the base layer obtain a single output signal having the mixed reality content; and

a display unit comprising a second semiconductor apparatus including a second substrate and second logic coupled to the second substrate, wherein the second logic includes:
a demultiplexer to receive a single complex signal having multiple interleaved signals and to separate one or more of a base layer signal, a first non-base layer signal, a second non-base layer signal, or a third non-base layer signal from the single complex signal;
a base layer decoder to decode the base layer signal into the real world content and a first layer decoder to decode the first non-base layer signal into a mixed reality content overlay;
a second layer decoder to decode the second non-base layer signal into a map overlay and a third layer decoder to decode the third non-base layer signal into a blend overlay; and
a compositor to composite one or more selected overlays with the decoded base layer signal for display.

14. The system of claim 13 , wherein the real world content includes captured video.

15. The system of claim 13 , wherein the map overlay distinguishes between placement of objects within an overlay auxiliary picture on another layer.

16. The system of claim 13 , wherein the blend overlay represents transparency of an overlay object from one or more overlay auxiliary pictures.

17. The system of claim 13 , wherein one or more of the base layer encoder, the first layer encoder, the second layer encoder, or the third layer encoder encode according to a High Efficiency Video Coding (HEVC) standard.

18. The system of claim 17 , wherein one or more of the base layer decoder, the first layer decoder, the second layer decoder, or the third layer decoder decode according to the High Efficiency Video Coding (HEVC) standard.

Assignees

• Intel Corp

Inventors

• Boyce Jill M
• Rowe Keith W
• Holland James M
• Fu Fangwen
• Yedidi Satya N
• Mohan Sumit

Classifications

• H04N19/30 Primary

  using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence) · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T2200/28

  involving image processing hardware · CPC title

• H04N21/234

  Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs · CPC title

• H04N19/27

  involving both synthetic and natural picture components, e.g. synthetic natural hybrid coding [SNHC] · CPC title