Image processing device and image processing method

Patent metadata

Field	Value
Publication number	US-10412413-B2
Application number	US-201615546364-A
Country	US
Kind code	B2
Filing date	Feb 17, 2016
Priority date	Mar 5, 2015
Publication date	Sep 10, 2019
Grant date	Sep 10, 2019

Abstract

There is provided an encoder that enables an image of a field-of-view range of a user to be displayed immediately. The encoder encodes, for a pair of opposite surfaces among a plurality of surfaces of a model for an omnidirectional image, a first composite image obtained by combining an omnidirectional image of a first surface of a high resolution and an omnidirectional image of a second surface of a resolution lower than the high resolution and a second composite image obtained by combining an omnidirectional image of the second surface of the high resolution and an omnidirectional image of the first surface of a low resolution. For example, the present disclosure can be applied to an image display system or the like.

First claim

The invention claimed is:

1. An image processing device, comprising:
a combining unit configured to generate, for a pair of opposite surfaces among a plurality of surfaces of a model for an omnidirectional image, a first composite image by combining the omnidirectional image of a first surface, of the pair of opposite surfaces, of a first resolution and the omnidirectional image of a second surface, of the pair of opposite surfaces, of a second resolution lower than the first resolution and a second composite image by combining the omnidirectional image of the second surface of the first resolution and the omnidirectional image of the first surface of the second resolution; and
an encoding unit configured to
encode, for generating an image of a field-of-view range of a user corresponding to a line-of-sight direction of the user, the first composite image and the second composite image, and
generate encoded streams of the first composite image and the second composite image,

wherein each of the first composite image and the second composite image include only images with respect to the first surface and the second surface, and
wherein the combining unit and the encoding unit are each implemented via at least one processor.

2. The image processing device according to claim 1 ,
wherein the encoding unit encodes the first composite image obtained by combining the omnidirectional image of the first surface of the first resolution, the omnidirectional image of the second surface of the second resolution, and a parallax image of the first surface of a third resolution and the second composite image obtained by combining the omnidirectional image of the second surface of the first resolution, the omnidirectional image of the first surface of the second resolution, and a parallax image of the second surface of the third resolution.

3. The image processing device according to claim 2 ,
wherein the encoding unit encodes a luma component of the first composite image obtained by combining a luma component of the omnidirectional image of the first surface of the first resolution, a luma component of the omnidirectional image of the second surface of the second resolution, and a parallax image of the first surface of the third resolution and a luma component of the second composite image obtained by combining a luma component of the omnidirectional image of the second surface of the first resolution, a luma component of the omnidirectional image of the first surface of the second resolution, and a parallax image of the second surface of the third resolution.

4. The image processing device according to claim 2 ,
wherein the second and third resolutions are ¼ of the first resolution.

5. The image processing device according to claim 2 ,
wherein the encoding unit encodes a chroma component of the first composite image obtained by combining a chroma component of the omnidirectional image of the first surface of the first resolution, a chroma component of the omnidirectional image of the second surface of the second resolution, and a parallax image of the second surface of the third resolution and a chroma component of the second composite image obtained by combining a chroma component of the omnidirectional image of the second surface of the first resolution, a chroma component of the omnidirectional image of the first surface of the second resolution, and a parallax image of the first surface of the third resolution.

6. The image processing device according to claim 1 , further comprising:
a generating unit configured to generate the first composite image and the second composite image,
wherein the generating unit is implemented via at least one processor.

7. The image processing device according to claim 1 , further comprising:
a converting unit configured to convert the resolutions of the omnidirectional images of the first surface and the second surface into the second resolution,
wherein the converting unit is implemented via at least one processor.

8. The image processing device according to claim 1 , further comprising:
a non-transitory computer-readable medium configured to store the encoded streams generated by the encoding unit; and
a delivering unit configured to deliver the encoded streams stored in the non-transitory computer-readable medium to a terminal that decodes the encoded streams,
wherein the delivering unit is implemented via at least one processor.

9. The image processing device according to claim 1 ,
wherein the encoding unit encodes the first composite image and the second composite image in accordance with an AVC/H.264 standard or an HEVC/H.265 standard.

10. The image processing device according to claim 1 ,
wherein the omnidirectional image is an image generated by cube mapping or an image generated by equidistant cylindrical projection.

11. The image processing device according to claim 1 ,
wherein the encoding unit is further configured to generate, for each pair of opposite surfaces among a plurality of pairs of opposite surfaces of the model for the omnidirectional image, the first composite image and the second composite image, a number of the composite images of the plurality of pairs of opposite surfaces being equal to a number of the plurality of surfaces of the model for the omnidirectional image.

12. An image processing method, comprising:
generating, by an image processing device, for a pair of opposite surfaces among a plurality of surfaces of a model for an omnidirectional image, a first composite image by combining the omnidirectional image of a first surface, of the pair of opposite surfaces, of a first resolution and the omnidirectional image of a second surface, of the pair of opposite surfaces, of a first resolution and the omnidirectional image of a second surface, of the pair of opposite surfaces, of a second resolution lower than the first resolution and a second composite image by combining the omnidirectional image of the second surface of the first resolution and the omnidirectional image of the first surface of the second resolution;
encoding, for generating an image of a field-of-view range of a user corresponding to a line-of-sight direction of the user, the first composite image and the second composite image; and
generating encoded streams of the first composite image and the second composite image,
wherein each of the first composite image and the second composite image include only images with respect to the first surface and the second surface.

13. An image processing device, comprising:
a decoding unit configured to decode, for a pair of opposite surfaces among a plurality of surfaces of a model for an omnidirectional image, an encoded stream obtained by combining the omnidirectional image of a first resolution of a surface corresponding to a line-of-sight direction of a user out of an encoded stream of a first composite image obtained by combining the omnidirectional image of a first surface, of the pair of opposite surfaces, of the first resolution and the omnidirectional image of a second surface, of the pair of opposite surfaces, of a second resolution lower than the first resolution and an encoded stream of a second composite image obtained by combining the omnidirectional image of the second surface of the first resolution and the omnidirectional image of the first surface of the second resolution; and
a rendering unit configured to generate an image of a field-of-view range of the user corresponding to the line-of-sight direction of the user by using the decoded streams,
wherein each of the first composite image and the second composite image include only images with respect to

Assignees

• Sony Corp
• Sony Interactive Entertainment Inc

Inventors

• Izumi Nobuaki
• Ando Yuji
• Shinohara Takayuki

Classifications

• H04N13/00

  Stereoscopic video systems; Multi-view video systems; Details thereof · CPC title

• G06T9/00

  Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) · CPC title

• H04N21/4402

  involving reformatting operations of video signals for household redistribution, storage or real-time display {(details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764)} · CPC title

• H04N21/2343

  involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements {(video transcoding H04N19/40; media packet handling at the source H04L65/762)} · CPC title

• H04N21/4728

  for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region · CPC title