Saliency information acquisition device and saliency information acquisition method

Patent metadata

Field	Value
Publication number	US-2016358035-A1
Application number	US-201615083018-A
Country	US
Kind code	A1
Filing date	Mar 28, 2016
Priority date	Jun 4, 2015
Publication date	Dec 8, 2016
Grant date	—

Abstract

A saliency information acquisition device has a local saliency acquisition unit configured to calculate a saliency measure for each pixel in an input image on the basis of information obtained from a local region surrounding each pixel, a candidate-region setting unit configured to set a plurality of candidate regions in the input image, a global saliency acquisition unit configured to calculate a saliency measure for each candidate region in the plurality of candidate regions on the basis of information including a local saliency feature representing an attribute of the saliency measure for each pixel within a candidate region, and a global feature representing an attribute of the candidate regions in relation to the entire input image, and an integration unit configured to combine the saliency measure for each candidate region in the plurality of candidate regions obtained by the global saliency acquisition unit to generate saliency information.

First claim

1 . A saliency information acquisition device comprising:
a local saliency acquisition unit configured to calculate a saliency measure for each pixel in an input image on the basis of information obtained from a local region surrounding each pixel; a candidate-region setting unit configured to set a plurality of candidate regions in the input image; a global saliency acquisition unit configured to calculate a saliency measure for each candidate region in the plurality of candidate regions on the basis of information including a local saliency feature representing an attribute of the saliency measure for each pixel within a candidate region, and a global feature representing an attribute of the candidate regions in relation to the entire input image; and an integration unit configured to combine the saliency measure for each candidate region in the plurality of candidate regions obtained by the global saliency acquisition unit to generate saliency information for the input image.

2 . The saliency information acquisition device according to claim 1 , wherein the local saliency acquisition unit uses a first deep neural network to calculate a saliency measure for each pixel in the input image.

3 . The saliency information acquisition device according to claim 2 , wherein the first deep neural network is a neural network configured to receive an image of a local region surrounding a given pixel as an input, and output a saliency measure for said pixel as an estimation result.

4 . The saliency information acquisition device according to claim 1 , further comprising:
a local saliency refinement unit configured to refine the saliency measure for each pixel on the basis of the saliency measure for each pixel obtained by the local saliency acquisition unit and the plurality of candidate regions obtained by the candidate-region setting unit so that the saliency measure for a pixel included in the candidate region is increased, and the saliency measure for a pixel outside of the candidate region is decreased, wherein the global saliency acquisition unit uses the saliency measure refined by the local saliency refinement unit to obtain the local saliency feature for each of the candidate regions.

5 . The saliency information acquisition device according to claim 1 , wherein the global saliency acquisition unit uses a second deep neural network to calculate a saliency measure for each candidate region in the plurality of candidate regions.

6 . The saliency information acquisition device according to claim 5 , wherein the second deep neural network is a neural network configured to receive as input a feature vector containing the local saliency feature and the global feature of a given candidate region as components, and output a saliency measure of said candidate region as an estimation result.

7 . The saliency information acquisition device according to claim 1 , wherein the local saliency feature includes one of:
a statistic of the saliency measure for each pixel within the candidate region; a ratio of the total of the saliency measure for each pixel in a candidate region to the total of the saliency measure for each pixel in the entire input image; a product of the statistic and the ratio; and an overlap ratio of the aforementioned candidate region to the regions in the input image where the saliency measure is nonzero.

8 . The saliency information acquisition device according to claim 1 , wherein the global feature includes one of:
a difference between a color histogram for the candidate region and the color histogram of an end region in the input image; a difference between a representative color value within the candidate region and a representative color value within the end region in the input image; a difference between a color histogram of the candidate region and a color histogram of the entire input image; and the dispersion of the colors within the candidate region.

9 . The saliency information acquisition device according to claim 1 , wherein a global feature includes one of:
the aspect ratio of a square enclosing the candidate region; the height of said square; the width of said square; the center coordinate of the candidate region; the length along the long axis of the candidate region; the length along the short axis of the candidate region; and an Euler number for the candidate region.

10 . The saliency information acquisition device according to claim 6 , wherein the integration unit is configured to perform a weighted sum of the saliency measures for the plurality of candidate regions using a weight corresponding to the reliability of the estimation result from the second deep neural network for each candidate region to generate saliency information for the input image.

11 . The saliency information acquisition device according to claim 1 , wherein the candidate-region setting unit detects a plurality of objectness regions from within the input image, and sets the plurality of detected regions as the plurality of candidate regions.

12 . The saliency information acquisition device according to claim 11 , wherein the candidate-region setting unit uses the Geodesic Object Proposal technique to detect the objectness regions from within the input image.

13 . A saliency information acquisition method having steps comprising:
calculating a saliency measure for each pixel in an input image on the basis of information obtained from a local region surrounding each pixel; setting a plurality of candidate regions in the input image; calculating a saliency measure for each candidate region in the plurality of candidate regions on the basis of information including a local saliency feature representing an attribute of the saliency measure for each pixel within a candidate region, and a global feature representing an attribute of the candidate regions in relation to the entire input image; and combining the saliency measure for each candidate region in the plurality of candidate regions to generate saliency information for the input image.

14 . The saliency information acquisition device according to claim 2 , further comprising:
a local saliency refinement unit configured to refine the saliency measure for each pixel on the basis of the saliency measure for each pixel obtained by the local saliency acquisition unit and the plurality of candidate regions obtained by the candidate-region setting unit so that the saliency measure for a pixel included in the candidate region is increased, and the saliency measure for a pixel outside of the candidate region is decreased, wherein the global saliency acquisition unit uses the saliency measure refined by the local saliency refinement unit to obtain the local saliency feature for each of the candidate regions.

15 . The saliency information acquisition device according to claim 3 , further comprising:
a local saliency refinement unit configured to refine the saliency measure for each pixel on the basis of the saliency measure for each pixel obtained by the local saliency acquisition unit and the plurality of candidate regions obtained by the candidate-region setting unit so that the saliency measure for a pixel included in the candidate region is increased, and the saliency measure for a pixel outside of the candidate region is decreased, wherein the global saliency acquisition unit uses the saliency measure refined by the local saliency refinement unit to obtain the local saliency feature for each of the candidate regions.

16 . The saliency information acquisition device according to claim 2 , wherein the global s

Assignees

• Omron Tateisi Electronics Co

Inventors

• Ruan Xiang
• Lu Huchuan

Classifications

• G06V10/454 Primary

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title

• G06F18/2415

  based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title