Foreground extraction apparatus, foreground extraction method, and recording medium

Patent metadata

Field	Value
Publication number	US-12307687-B2
Application number	US-201917783715-A
Country	US
Kind code	B2
Filing date	Dec 16, 2019
Priority date	Dec 16, 2019
Publication date	May 20, 2025
Grant date	May 20, 2025

Abstract

In a foreground extraction apparatus, an extraction result generation unit performs a foreground extraction using a plurality of foreground extraction models for an input image, and generates foreground extraction results. A selection unit selects one or more foreground extraction models among the plurality of foreground extraction models using respective foreground results acquired by the plurality of foreground extraction models. A foreground region generation unit extracts each foreground region based on the input image using the selected one or more foreground extraction models.

First claim

What is claimed is:

1. A foreground extraction apparatus comprising:
a memory storing instructions; and
one or more processors configured to execute the instructions to:
generate respective foreground extraction results by performing a foreground extraction using a plurality of foreground extraction models with respect to an input image;
select one or more foreground extraction models from among the plurality of foreground extraction models by using the respective foreground extraction results of the plurality of foreground extraction models; and
extract each foreground region based on the input image by using the selected one or more foreground extraction models,
wherein the one or more processors:
generate time series features respectively for the plurality of foreground extraction models based on the input image, and select the foreground extraction model based on the generated time series;
generate each tracker based on the input image, calculate respective distances between each tracker and each foreground included in the respective foreground extraction results, and generate the time series features using the respective distances; and
calculate foreground degrees of foregrounds included in the foreground extraction result, and generate the time series features using the foreground degrees,

wherein the time series features are indicated by values each of which is acquired by dividing a total value, which is obtained by dividing the distances by respective foreground degrees for all trackers and all foregrounds, by a number of the trackers, and
wherein the one or more processors:
select a foreground extraction model having a minimum value among the time series features.

2. The foreground extraction apparatus according to claim 1 , wherein the one or more processors are further configured to
calculate a time series feature corresponding to each foreground region generated by the foreground region generation unit, and
identify an object corresponding to each foreground region based on the input image and the time series feature calculated by the time series feature calculation unit.

3. The foreground extraction apparatus according to claim 1 , wherein the one or more processors select the foreground extraction models by using a selection model which is trained by the input image and correct answer data of the foreground extraction results.

4. The foreground extraction apparatus according to claim 3 , wherein
the selection model is trained in order for a value Q(s, a) to be greater, in which a state s denotes a feature vector extracted from the input image, a reward r denotes a difference between the foreground extraction result and the correct answer data, and the value Q denotes an action a for selecting any of the plurality of foreground extraction models in the state s, and
the one or more processors select the foreground extraction model based on the value Q(s, a).

5. The foreground extraction apparatus according to claim 4 , wherein the value Q(s, a) is given by:

Q
⁡
(

s

t
+
1

,

a

t
+
1

)

←

Q
⁡
(

s
t

,

a
t

)

+

α
⁡
(

r

t
+
1

+

γ

max
a

Q
⁡
(

s

t
+
1

,
a

)

-

Q
⁡
(

s
t

,

a
t

)

)

.

6. The foreground extraction apparatus according to claim 3 , wherein
the selection model is trained in order for a difference between a prediction result and the correct answer data to be smaller, in which the prediction result is represented by a weighted sum of respective weights for the foreground extraction models calculated by a likelihood estimator based on the input image and respective foreground extraction results of the foreground extraction models, and
the one or more processors select the foreground extraction models based on the respective weights for the foreground extraction models.

7. The foreground extraction apparatus according to claim 3 , wherein
the selection model is trained in order for a difference between a prediction result and the correct answer data to be smaller, in which the prediction result is represented by summing products of respective spatial maps of the foreground extraction models calculated by a likelihood estimator based on the input image and the respective foreground extraction results of the foreground extraction models, and
the one or more processors select the foreground extraction models based on the respective spatial maps for the foreground extraction models.

8. A foreground extraction method performed by a computer and comprising:
generating respective foreground extraction results by performing a foreground extraction using a plurality of foreground extraction models with respect to an input image;
selecting one or more foreground extraction models from among the plurality of foreground extraction models by using the respective foreground extraction results of the plurality of foreground extraction models; and
extracting each foreground region based on the input image by using the selected one or more foreground extraction models,
wherein the computer:
generates time series features respectively for the plural

Assignees

• Nec Corp

Inventors

• Sato Hideaki
• Higa Kyota
• Babazaki Yasunori
• Miyano Hiroyoshi

Classifications

• G06T2207/10016

  Video; Image sequence · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/267

  by performing operations on regions, e.g. growing, shrinking or watersheds · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G06T7/215

  Motion-based segmentation · CPC title