Image processing apparatus, image processing method, and storage medium

Patent metadata

Field	Value
Publication number	US-11778155-B2
Application number	US-202117522508-A
Country	US
Kind code	B2
Filing date	Nov 9, 2021
Priority date	Nov 11, 2020
Publication date	Oct 3, 2023
Grant date	Oct 3, 2023

Abstract

An image processing apparatus includes an acquisition unit configured to acquire line-of-sight information about a movement range of a line of sight of a subject imaged by a plurality of image capturing apparatuses, a determination unit configured to determine virtual viewpoint information indicating a virtual viewpoint related to a virtual viewpoint image that is based on a plurality of captured images acquired by the imaging by the plurality of image capturing apparatuses and that includes the movement range of the line of sight of the subject identified based on the line-of-sight information acquired by the acquisition unit, and an output unit configured to output the virtual viewpoint information determined by the determination unit.

First claim

What is claimed is:

1. An image processing apparatus comprising:
one or more memories; and
one or more processors in communication with the one or more memories, wherein the one or more processors and the one or more memories are configured to:
acquire line-of-sight information about a movement range of a line of sight of a subject imaged by a plurality of image capturing apparatuses, based on a movement locus of the line of sight of the subject in a predetermined time range;
determine virtual viewpoint information indicating a virtual viewpoint related to a virtual viewpoint image that is based on a plurality of captured images acquired by the imaging by the plurality of image capturing apparatuses and that includes the movement range of the line of sight of the subject identified based on the line-of-sight information; and
output the virtual viewpoint information,
wherein in a case where the virtual viewpoint information includes information that causes change of a visual field size of the virtual viewpoint to a narrow angle, the one or more processors and the one or more memories are further configured to gradually change the visual field size, and
wherein the one or more processors and the one or more memories are further configured to acquire the line-of-sight information about the movement range of the line of sight, based on a plurality of the movement loci of the line of sight that is respectively acquired in a first predetermined time range and a second predetermined time range each of which is the predetermined time range.

2. The image processing apparatus according to claim 1 ,
wherein the one or more processors and the one or more memories are further configured to determine the virtual viewpoint information based on the line-of-sight information acquired based on the plurality of movement loci of the line of sight.

3. The image processing apparatus according to claim 2 , wherein the one or more processors and the one or more memories are further configured to gradually change the virtual viewpoint information between first virtual viewpoint information determined to correspond to the first predetermined time range and second virtual viewpoint information determined to correspond to the second predetermined time range that is shifted by the predetermined time range from the first predetermined time range.

4. The image processing apparatus according to claim 3 , wherein the one or more processors and the one or more memories are further configured to determine the first virtual viewpoint information and the second virtual viewpoint information as the virtual viewpoint information for key frames, and determine the virtual viewpoint information to be complemented for each time obtained by dividing the predetermined time range by a number of frames for displaying the virtual viewpoint image, as the virtual viewpoint information to be changed gradually.

5. The image processing apparatus according to claim 1 , wherein the one or more processors and the one or more memories are further configured to acquire an instruction about the predetermined time range, and update the predetermined time range based on the instruction.

6. The image processing apparatus according to claim 1 , wherein the one or more processors and the one or more memories are further configured to perform control so that an image related to the line of sight of the subject is superimposed on the virtual viewpoint image corresponding to the virtual viewpoint information.

7. The image processing apparatus according to claim 6 , wherein the one or more processors and the one or more memories are further configured to perform control so that an image indicating a center of the line of sight of the subject is superimposed as the image related to the line of sight of the subject.

8. The image processing apparatus according to claim 6 , wherein the one or more processors and the one or more memories are further configured to perform control so that the image related to the line of sight of the subject and superimposed on the virtual viewpoint image gradually disappears with passage of time.

9. The image processing apparatus according to claim 1 , wherein the virtual viewpoint information includes at least one of information indicating a position of the virtual viewpoint, information indicating a line-of-sight direction from the virtual viewpoint, or information indicating a visual field size of the virtual viewpoint.

10. The image processing apparatus according to claim 1 , wherein the movement range of the line of sight is included within an effective visual field range, excluding a peripheral visual field range, of the line of sight of the subject.

11. The image processing apparatus according to claim 1 , wherein the subject is a player or a referee imaged by the plurality of image capturing apparatuses.

12. The image processing apparatus according to claim 1 , wherein the one or more processors and the one or more memories are further configured to generate the virtual viewpoint image based on the virtual viewpoint information.

13. An image processing method comprising:
acquiring line-of-sight information about a movement range of a line of sight of a subject imaged by a plurality of image capturing apparatuses, based on a movement locus of the line of sight of the subject in a predetermined time range;
determining virtual viewpoint information indicating a virtual viewpoint related to a virtual viewpoint image that is based on a plurality of captured images acquired by the imaging by the plurality of image capturing apparatuses and that includes the movement range of the line of sight of the subject identified based on the acquired line-of-sight information; and
outputting the determined virtual viewpoint information,
wherein in a case where the virtual viewpoint information includes information that causes change of a visual field size of the virtual viewpoint to a narrow angle, the determining gradually changes the visual field size, and
wherein the one or more processors and the one or more memories are further configured to acquire the line-of-sight information about the movement range of the line of sight, based on a plurality of the movement loci of the line of sight that is respectively acquired in a first predetermined time range and a second predetermined time range each of which is the predetermined time range.

14. A non-transitory computer-readable storage medium storing a program for causing a computer to execute an image processing method, the method comprising:
acquiring line-of-sight information about a movement range of a line of sight of a subject imaged by a plurality of image capturing apparatuses, based on a movement locus of the line of sight of the subject in a predetermined time range;
determining virtual viewpoint information indicating a virtual viewpoint related to a virtual viewpoint image that is based on a plurality of captured images acquired by the imaging by the plurality of image capturing apparatuses and that includes the movement range of the line of sight of the subject identified based on the acquired line-of-sight information; and
outputting the determined virtual viewpoint information,
wherein in a case where the virtual viewpoint information includes information that causes change of a visual field size of the virtual viewpoint to a narrow angle, the determining gradually changes the visual field size, and
wherein the one or more processors and the one or more memories are further configured to acquire the line-of-sight information about the movement range of the line of sight, based on a plurality of the movement loci of

Assignees

• Canon Kk

Inventors

• Yano Tomohiro

Classifications

• H04N13/111 Primary

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• H04N5/272

  Means for inserting a foreground image in a background image, i.e. inlay, outlay · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06T2207/30221

  Sports video; Sports image · CPC title