Method for generating virtual viewpoint image and image processing apparatus

Patent metadata

Field	Value
Publication number	US-2019356906-A1
Application number	US-201716303477-A
Country	US
Kind code	A1
Filing date	May 22, 2017
Priority date	May 25, 2016
Publication date	Nov 21, 2019
Grant date	—

Abstract

A method for generating a virtual viewpoint image includes generating, by a first image processing apparatus, first information for generating the virtual viewpoint image based on a first image captured by a first camera, generating, by a second image processing apparatus, second information for generating the virtual viewpoint image based on a second image captured by a second camera; specifying a viewpoint in the virtual viewpoint image; and generating, by a third image processing apparatus, a virtual viewpoint image corresponding to the specified viewpoint using the first information and the second information.

First claim

1 - 15 . (canceled)

16 . An image processing system, comprising:
a plurality of cameras configured to capture an imaging target region from a plurality of directions; a plurality of image processing apparatuses including a first image processing apparatus which extracts an object region from a first image captured by a first camera in the plurality of cameras and a second image processing apparatus which extracts an object region from a second image captured by a second camera in the plurality of cameras; and an image generating apparatus configured to generate a virtual viewpoint image based on results of the extraction of the object regions performed by the plurality of image processing apparatuses, wherein the plurality of image processing apparatuses are connected to each other through a daisy chain, and wherein image data based on the results of the extraction of the object regions performed by the first and second image processing apparatuses is transmitted to the image generating apparatus in accordance with the daisy chain connection between the plurality of image processing apparatuses.

17 . The image processing system according to claim 16 ,
wherein at least one of the first and second image processing apparatuses extracts a non-object region which is at least different from the object region from the image captured by the corresponding one of the cameras, and wherein at least one of the image processing apparatuses compresses the image data corresponding to the result of the extraction of the object region and image data corresponding to a result of the extraction of the non-object region such that a compression rate of the image data corresponding to the result of the extraction of the object region is lower than a compression rate of the image data corresponding to the result of the extraction of the non-object region and transmits the image data.

18 . The image processing system according to claim 16 ,
wherein at least one of the first and second image processing apparatuses extracts a non-object region which is at least different from the object region from the image captured by the corresponding one of the cameras, and wherein at least one of the image processing apparatuses transmits image data corresponding to a result of the extraction of the object region without compression and transmits image data corresponding to a result of the extraction of the non-object region after the image data is compressed.

19 . The image processing system according to claim 16 ,
wherein at least one of the first and second image processing apparatuses extracts a non-object region which is at least different from the object region from the image captured by the corresponding one of the cameras, and wherein at least one of the image processing apparatuses transmits image data corresponding to a result of the extraction of the object region and image data corresponding to a result of the extraction of the non-object region such that transmission of the image data corresponding to the result of the extraction of the non-object region is less frequently that transmission of the image data corresponding to the result of the extraction of the object region.

20 . The image processing system according to claim 17 , wherein the image data corresponding to the result of the extraction of the object region and the image data corresponding to the result of the extraction of the non-object region are transmitted to one of the plurality of image processing apparatuses which is different from the at least one of the image processing apparatuses or the image generating apparatus.

21 . The image processing system according to claim 17 , wherein the image data corresponding to the result of the extraction of the non-object region is image data of the non-object region.

22 . The image processing system according to claim 16 , wherein one of the plurality of image processing apparatuses which is different from a first one of the image processing apparatuses in predetermined order generates image data to be transmitted to a next one of the image processing apparatuses in the predetermined order based on image data corresponding to a result of the extraction of the object region and image data corresponding to a result of extraction of an object region performed by a preceding one of the image processing apparatuses in the predetermined order.

23 . The image processing system according to claim 16 , wherein the image generating apparatus comprising:
a reception unit configured to receive information on a virtual viewpoint; and an image generating unit configured to generate the virtual viewpoint image based on a position and an orientation of the virtual viewpoint based on the information received by the reception unit.

24 . The image processing system according to claim 23 , wherein the image generating apparatus further comprising:
a data generating unit configured to generate three-dimensional shape data of objects based on image data corresponding to results of the extraction of the object regions performed by the plurality of image processing apparatuses, and wherein the image generating unit generates a virtual viewpoint image based on the three-dimensional shape data generated by the data generating unit and the position and the orientation of the virtual viewpoint corresponding to the information received by the reception unit.

25 . The image processing system according to claim 16 , further comprising a synchronization unit configured to synchronize imaging timings of the plurality of cameras.

26 . The image processing system according to claim 16 , wherein a ratio of the plurality of cameras to the plurality of image processing apparatuses is N to M (N and M are integers not less than 1).

27 . The image processing system according to claim 16 , wherein the image data corresponding to a result of the extraction of the object region is image data of the object region.

28 . The image processing system according to claim 16 , wherein the object region includes a region of a moving body.

29 . The image processing system according to claim 16 , wherein at least one of a region of a person and a region of a ball is included in the object region.

30 . An image processing apparatus, comprising:
an object extraction unit configured to extract an object region from an image captured by a camera; an obtaining unit configured to obtain, from another image processing apparatus, image data corresponding to an object region extracted by the other image processing apparatus from an image captured by another camera; and a transmission unit configured to transmit generation image data based on image data corresponding to a result of the extraction of the object region performed by the object extraction unit and image data obtained from the other image processing apparatus by the obtaining unit to an image generating apparatus which generates a virtual viewpoint image.

31 . The image processing apparatus according to claim 30 , further comprising:
a non-object extraction unit configured to extract a non-object region which is at least different from the object region from an image captured by the camera, wherein the transmission unit compresses the generation image data and image data corresponding to a result of the extraction of the non-object region extracted by the non-object extraction unit such that a compression rate of the generation image data is lower than a compression rate of the image data corresponding to a result of the extraction of the non-object region and tra

Assignees

• Canon Kk

Inventors

• Handa Masahiro
• Aizawa Michio
• Mizuno Shogo
• Tanaka Katsumasa
• Matsushita Akihiro
• Morisawa Keisuke
• Yano Tomohiro
• Komiyama Mai
• Fujii Kenichi
• Date Atsushi

Classifications

• H04N13/00

  Stereoscopic video systems; Multi-view video systems; Details thereof · CPC title

• H04N13/296

  Synchronisation thereof; Control thereof · CPC title

• G06T7/73

  using feature-based methods · CPC title

• H04N13/117 Primary

  the virtual viewpoint locations being selected by the viewers or determined by viewer tracking · CPC title

• H04N13/239

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title