Adapting video images for wearable devices

Patent metadata

Field	Value
Publication number	US-2018295352-A1
Application number	US-201815949720-A
Country	US
Kind code	A1
Filing date	Apr 10, 2018
Priority date	Apr 11, 2017
Publication date	Oct 11, 2018
Grant date	—

Abstract

A spatial direction of a wearable device that represents an actual viewing direction of the wearable device is determined. The spatial direction of the wearable device is used to select, from a multi-view image comprising single-view images, a set of single-view images. A display image is caused to be rendered on a device display of the wearable device. The display image represents a single-view image as viewed from the actual viewing direction of the wearable device. The display image is constructed based on the spatial direction of the wearable device and the set of single-view images.

First claim

What is claimed is:

1 . A method, comprising:
determining a spatial direction of a wearable device, the spatial direction of the wearable device representing an actual viewing direction of the wearable device at a first time point; using the spatial direction of the wearable device that represents the actual viewing direction of the wearable device to select, from a multi-view image comprising a plurality of single-view images, a set of two or more single-view images corresponding to a set of two or more viewing directions at the first time point, each single-view image in the plurality of single-view images in the multi-view image (a) corresponding to a respective viewing direction in a plurality of viewing directions and (b) representing a view of the multi-view image from the respective viewing direction; causing a display image to be rendered on a device display of the wearable device, the display image representing a single-view image as viewed from the actual viewing direction of the wearable device at the first time point, the display image being constructed based at least in part on the spatial direction of the wearable device and the set of two or more single-view images corresponding to the set of two or more viewing directions.

2 . The method of claim 1 , wherein the plurality of single-view images of the multi-view image is received with depth information for the plurality of single-view images, and wherein the display images are constructed based further on the depth information.

3 . The method of claim 1 , wherein the spatial direction of the wearable device is determined based on one or more spatial coordinates of the wearable device, and wherein the one or more spatial coordinates of the wearable device comprise one of: one or more rotational coordinates only, or a combination of translational coordinates and rotational coordinates.

4 . The method of claim 1 , wherein single-view images in the set of two or more single-view images are selected based further on a spatial position of the wearable device.

5 . The method of claim 1 , wherein the spatial direction of the wearable device is determined in relation to a reference coordinate system of a 3D space in which the wearable device resides.

6 . The method of claim 1 , wherein the plurality of viewing directions supported by the plurality of single-view images of the multi-view image forms a solid angle in relation to a viewer of the wearable device.

7 . The method of claim 1 , wherein the display image comprises a left view display image and a right view display image that form a stereoscopic image.

8 . The method of claim 1 , wherein a cinema display image depicting a first proper subset of visual objects in a plurality of visual objects in a 3D image space; wherein the display image represents a device display image that depicts one or more second proper subsets of visual objects in the plurality of visual objects in the 3D image space; and wherein the cinema display image is concurrently rendered on a cinema display for viewing by a viewer while the display image is rendered on the device display of the wearable device for viewing by the same viewer.

9 . The method of claim 8 , wherein the cinema display image depicting the first proper subset of visual objects in the plurality of visual objects in the 3D image space is received in a cinema image layer of a multi-layer multi-view video signal, and wherein the image set used to construct the display image is received in one or more device image layers of the multi-layer multi-view video signal.

10 . The method of claim 8 , further comprising:
receiving timing information associated with the cinema display image; using the timing information to synchronize rendering the cinema display image and the device display image.

11 . The method of claim 1 , wherein the display image is constructed by interpolating single-view images in the image set based on the spatial direction of the wearable device.

12 . The method of claim 1 , wherein the image set includes a subset of relatively low resolution images each of which covers a relatively large field of view and a subset of relatively high resolution images each of which covers a relatively small focus region of a viewer.

13 . The method of claim 1 , wherein the display image is constructed by the wearable device.

14 . The method of claim 1 , wherein the display image is constructed by a video streaming server that streams the display image to the wearable device.

15 . The method of claim 1 , wherein the image set is transmitted from a video streaming server to the wearable device.

16 . The method of claim 1 , wherein the plurality of single-view images is received locally from a storage device accessible to the wearable device.

17 . A method, comprising:
determining a spatial direction of a wearable device, the spatial direction of the wearable device representing an actual viewing direction of the wearable device at a first time point; receiving a set of two or more single-view images corresponding to a set of two or more viewing directions at the first time point, the spatial direction of the wearable device being used to select, from a multi-view image comprising a plurality of single-view images, single-view images into the set of two or more single-view images, each single-view image in the plurality of single-view images in the multi-view image (a) corresponding to a respective viewing direction in a plurality of viewing directions and (b) representing a view of the multi-view image from the respective viewing direction; constructing a display image based at least in part on the spatial direction of the wearable device and the set of two or more single-view images corresponding to the set of two or more viewing directions, the display image being rendered on a device display of the wearable device, the display image representing a single-view image as viewed from the actual viewing direction of the wearable device at the first time point.

18 . An apparatus performing the method as recited in claim 17 .

19 . A system performing the method as recited in claim 17 .

20 . A non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors cause performance of the method recited in claim 17 .

21 . A computing device comprising one or more processors and one or more storage media, storing a set of instructions, which when executed by one or more processors cause performance of the method recited in claim 17 .

Assignees

• Dolby Laboratories Licensing Corp

Inventors

• Ninan Ajit
• Mammen Neil

Classifications

• G06F3/012 Primary

  Head tracking input arrangements · CPC title

• H04N13/383

  for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes · CPC title

• H04N13/344 Primary

  with head-mounted left-right displays · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title