Image processing apparatus, display method, and non-transitory computer-readable storage medium

Patent metadata

Field	Value
Publication number	US-2021005023-A1
Application number	US-202016918954-A
Country	US
Kind code	A1
Filing date	Jul 1, 2020
Priority date	Jul 5, 2019
Publication date	Jan 7, 2021
Grant date	—

Abstract

A trajectory generation management unit determines a target foreground object based on a user's input operation for selecting a target foreground object. The trajectory generation management unit determines a time range to generate a trajectory image based on the user operation. A trajectory calculating unit calculates a trajectory of the target foreground object in the determined time range. A virtual viewpoint determining unit determines virtual viewpoint information based on the trajectory. An image generating unit determines plot times in the time range based on a user's plotting operation. The image generating unit generates a trajectory image based on the determined plot times.

First claim

What is claimed is:

1 . An image processing apparatus comprising:
an obtaining unit configured to obtain three dimensional shape data of an object, the three dimensional shape data being generated based on multiple images captured by multiple image capturing apparatuses; a determining unit configured to determine a line-of-sight direction of a virtual viewpoint of a virtual viewpoint image that is generated based on the multiple captured images; and a generating unit configured to, based on the line-of-sight direction of the virtual viewpoint, determined by the determining unit, and multiple pieces of the three dimensional shape data that are obtained by the obtaining unit and that respectively represent images of the object at different points in time in a period during Which images of the object are continuously captured by the multiple image capturing apparatuses, generate a virtual viewpoint image containing the images of the object at the different points in time.

2 . The image processing apparatus according to claim 1 , further comprising:
a time specifying unit configured to specify a time range including the different points in time, wherein the generating unit is configured to generate a virtual viewpoint image containing multiple pieces of three dimensional shape data at different points in time in the time range specified by the time specifying unit.

3 . The image processing apparatus according to claim 2 , wherein
the time specifying unit is configured to specify the time range based on a user operation for designating at least any one of start point time and end point time.

4 . The image processing apparatus according to claim 2 , wherein
the time specifying unit is configured to specify the time range based on a recognition result of at least any one of speech recognition and image recognition.

5 . The image processing apparatus according to claim 4 , wherein
the time range is a time range associated with a scene in which a predetermined event has occurred in an image capturing area of which images are captured by the multiple image capturing apparatuses.

6 . The image processing apparatus according to claim 1 , further comprising:
an object specifying unit configured to specify one or more objects from among multiple objects of which images are captured by the multiple image capturing apparatuses, wherein the obtaining unit is configured to obtain shape data that represents the one or more objects specified by the object specifying unit, and the generating unit is configured to, based on the line-of-sight direction of the virtual viewpoint, determined by the determining unit, and multiple pieces of the three dimensional shape data that are obtained by the obtaining unit, generate a virtual viewpoint image containing images of each of the one or more objects at the different points in time.

7 . The image processing apparatus according to claim 6 , wherein
the object specifying unit is configured to specify the one or more objects based on a user operation.

8 . The image processing apparatus according to claim 1 , wherein
the different points in time are designated based on a user operation.

9 . The image processing apparatus according to claim 1 , wherein
the different points in time are specified by dividing the period, during which images of the object are continuously captured, into multiple periods having a predetermined time length.

10 . The image processing apparatus according to claim 1 , wherein
the determining unit is configured to determine the line-of-sight direction of the virtual viewpoint based on locations of the object at the different points in time.

11 . The image processing apparatus according to claim 10 , wherein
the determining unit is configured to determine the line-of-sight direction of the virtual viewpoint such that the locations of the object at the different points in time are included in an angle of view.

12 . The image processing apparatus according to claim 10 , wherein
the determining unit is configured to determine the line-of-sight direction of the virtual viewpoint based on a line segment connecting the locations of the object at the different points in time.

13 . The image processing apparatus according to claim 1 , wherein
the generating unit is configured to generate a moving image based on the line-of-sight direction of the virtual viewpoint, determined by the determining unit, and the images captured during the period during which images are continuously captured and generate a superimposed moving image in which the virtual viewpoint image and the moving image are superimposed on each other.

14 . The image processing apparatus according to claim 1 , wherein
the generating unit is configured to generate a virtual viewpoint image containing an image of the object at specific time with highlight among the images of the object at the different points in time.

15 . A display method comprising:
determining a line-of-sight direction of a virtual viewpoint of a virtual viewpoint image based on multiple images captured by multiple image capturing apparatuses; and displaying, based on the determined line-of-sight direction of the virtual viewpoint and multiple pieces of three dimensional shape data that respectively represent images of an object at different points in time in a period during Which images of the object are continuously captured by the multiple image capturing apparatuses, a virtual viewpoint image containing the images of the object at the different points in time.

16 . A non-transitory computer-readable storage medium storing a program for causing a computer to execute a display method, the method comprising:
determining a line-of-sight direction of a virtual viewpoint of a virtual viewpoint image based on multiple images captured by multiple image capturing apparatuses; and displaying, based on the determined line-of-sight direction of the virtual viewpoint and multiple pieces of three dimensional shape data that respectively represent images of an object at different points in time in a period during Which images of the object are continuously captured by the multiple image capturing apparatuses, a virtual viewpoint image containing the images of the object at the different points in time.

Assignees

• Canon Kk

Inventors

• Yoshimura Yuri

Classifications

• G06T15/205

  Image-based rendering · CPC title

• H04N13/282

  for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems · CPC title

• H04N13/117 Primary

  the virtual viewpoint locations being selected by the viewers or determined by viewer tracking · CPC title

• H04N13/296

  Synchronisation thereof; Control thereof · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title