Methods and apparatus for requesting, receiving and/or playing back content corresponding to an environment

Patent metadata

Field	Value
Publication number	US-10742948-B2
Application number	US-201816040184-A
Country	US
Kind code	B2
Filing date	Jul 19, 2018
Priority date	Mar 1, 2015
Publication date	Aug 11, 2020
Grant date	Aug 11, 2020

Abstract

Methods and apparatus for receiving content including images of surfaces of an environment visible from a default viewing position and images of surfaces not visible from the default viewing position, e.g., occluded surfaces, are described. Occluded and non-occluded image portions are received in content streams that can be in a variety of stream formats. In one stream format non-occluded image content is packed into a frame with occluded image content with the occluded image content normally occupying a small portion of the frame. In other embodiments occluded image portions are received in an auxiliary data stream which is multiplexed with a data stream providing frames of non-occluded image content. UV maps which are used to map received image content to segments of an environmental model are also supplied with the UV maps corresponding to the format of the frames which are used to provide the images that serve as textures.

First claim

What is claimed:

1. A method of operating a content playback device, the method comprising:
receiving a first frame including non-occluded image content corresponding to a portion of an environment visible from a first location in the environment, said first location corresponding to a default location;
receiving an auxiliary frame including occluded image content corresponding to a portion of the environment which is not visible from said first location;
detecting a first head location of a user, said first head location being offset left or right, up or down from said default location;
rendering an image by: i) using a first UV map to map image portions included in the first frame to a first set of segments of a model of the environment and ii) using an auxiliary UV map to map image portions included in the auxiliary frame to a second set of segments of the model of the environment, said rendered image including a portion of the environment which would be visible to the user in the environment having the detected first head location, said image including at least some non-occluded image content from the first frame and some occlude image content from said auxiliary frame; and
outputting to the rendered image to a display.

2. The method of claim 1 , wherein the first location in the environment corresponds to a default head position, the method further comprising:
outputting to the display image portions corresponding to portions of the environment which are visible from the first location when the head location of the user is said default location.

3. The method of claim 1 , further comprising:
receiving the first UV map indicating a mapping between portions of the first frame including non-occluded image content to segments of a first model of the environment which correspond to portions of said environment visible from said first location; and
receiving the auxiliary UV map indicating a mapping between portions of the auxiliary frame including occluded image content to segments of the first model of the environment which correspond to portions of said environment which are not visible from said first location.

4. The method of claim 3 , further comprising:
receiving said first frame and auxiliary frame in a multiplexed content stream including a primary content stream including said first frame and an auxiliary content stream including said auxiliary frame.

5. The method of claim 4 , further comprising:
demultiplexing said primary content stream and said auxiliary content stream.

6. The method of claim 5 , wherein said first frame and said auxiliary frame are received in encoded form, the method further comprising:
using a first decoder to decode said first frame prior to image rendering; and
using a second decoder which is different from said first decoder to decode the said auxiliary frame prior to image rendering.

7. The method of claim 6 , wherein said auxiliary frame is the same size and includes the same number of pixels as said first frame.

8. The method of claim 6 , wherein said first frame and said auxiliary frame are received in encoded form, the method further comprising:
using a decoder on a time shared basis to decode said first frame and said auxiliary frame prior to image rendering.

Assignees

• Nextvr Inc

Inventors

• Cole David
• Moss Alan

Classifications

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title

• H04N13/232 Primary

  using fly-eye lenses, e.g. arrangements of circular lenses · CPC title

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• H04N13/282

  for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems · CPC title

• G09G2352/00

  Parallel handling of streams of display data · CPC title