Automatic image annotations

Patent metadata

Field	Value
Publication number	US-11615618-B2
Application number	US-202117225165-A
Country	US
Kind code	B2
Filing date	Apr 8, 2021
Priority date	Apr 8, 2021
Publication date	Mar 28, 2023
Grant date	Mar 28, 2023

Abstract

A computer-implemented method for annotating images is disclosed. The computer-implemented method includes generating a saliency map corresponding to an input image, wherein the input image is an image that requires annotation, generating a behavior saliency map, wherein the behavior saliency map is a saliency map formed from an average of a plurality of objects contained within respective bounding boxes of a plurality of sample images, generating a historical saliency map, wherein the historical saliency map is a saliency map formed from an average of a plurality of tagged objects in the plurality of sample images, fusing the saliency map corresponding to the input image, the behavior saliency map, and the historical saliency map to form a fused saliency map, and generating, based on the fused saliency map, a bounding box around an object in the input image.

First claim

What is claimed is:

1. A computer-implemented method for annotating images, comprising:
generating a saliency map corresponding to an input image, wherein the input image is an image that requires annotation;
generating a behavior saliency map, wherein the behavior saliency map is a saliency map formed from an average of a plurality of objects contained within respective bounding boxes of a plurality of sample images;
generating a historical saliency map, wherein the historical saliency map is a saliency map formed from an average of a plurality of tagged objects in the plurality of sample images;
fusing the saliency map corresponding to the input image, the behavior saliency map, and the historical saliency map to form a fused saliency map; and
generating, based on the fused saliency map, a bounding box around an object in the input image.

2. The computer-implemented method of claim 1 , wherein fusing the saliency map corresponding to the input image, the behavior saliency map, and the historical saliency map to form the fused saliency map is further based on, at least in part, on:
assigning one or more weighted values to the saliency map corresponding to the input image, the behavior saliency map, and historical saliency map, respectively.

3. The computer-implemented method of claim 2 , wherein the one or more weighted values assigned to the saliency map corresponding to the input image, the behavior saliency map, and the historical saliency map are automatically determined based on an image attention mechanism.

4. The computer-implemented method of claim 1 , wherein generating the saliency map corresponding to the input image further includes:
generating a hierarchical image layer; and
generating one or more saliency cues for the hierarchical image layer.

5. The computer-implemented method of claim 1 , wherein generating the historical saliency map further includes:
selecting one or more sample images above a predetermined threshold, wherein the predetermined threshold is based on a degree of similarity between the sample image and the input image.

6. The computer-implemented method of claim 1 , further comprising:
tagging the bounding box as a particular type of object.

7. The computer-implemented method of claim 6 , wherein tagging the bounding box includes comparing a first salient region contained within the bounding box of the fused saliency map to a second salient region of an image previously tagged.

8. A computer program for annotating images, the computer program product comprising one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the program instructions including instructions to:
generate a saliency map corresponding to an input image, wherein the input image is an image that requires annotation;
generate a behavior saliency map, wherein the behavior saliency map is a saliency map formed from an average of a plurality of objects contained within respective bounding boxes of a plurality of sample images;
generate a historical saliency map, wherein the historical saliency map is a saliency map formed from an average of a plurality of tagged objects in the plurality of sample images;
fuse the saliency map corresponding to the input image, the behavior saliency map, and the historical saliency map to form a fused saliency map; and
generate, based on the fused saliency map, a bounding box around an object in the input image.

9. The computer program product of claim 8 , wherein the instructions to fuse the saliency map corresponding to the input image, the behavior saliency map, and the historical saliency map to form the fused saliency map is further based on, at least in part, on instructions to:
assign one or more weighted values to the saliency map corresponding to the input image, the behavior saliency map, and historical saliency map, respectively.

10. The computer program product of claim 9 , wherein the instructions for the one or more weighted values assigned to the saliency map corresponding to the input image, the behavior saliency map, and the historical saliency map are automatically determined based on an image attention mechanism.

11. The computer program product of claim 8 , wherein the instructions to generate the saliency map corresponding to the input image further includes instructions to:
generate a hierarchical image layer; and
generate one or more saliency cues for the hierarchical image layer.

12. The computer program product of claim 8 , wherein the instructions to generate the historical saliency map further includes instructions to:
select one or more sample images above a predetermined threshold, wherein the predetermined threshold is based on a degree of similarity between the sample image and the input image.

13. The computer program product of claim 8 , further comprising instructions to:
tag the bounding box as a particular type of object.

14. The computer program product of claim 13 , wherein instructions to tag the bounding box further includes instructions to compare a first salient region contained within the bounding box of the fused saliency map to a second salient region of an image previously tagged.

15. A computer system for updating a device, comprising:
one or more computer processors;
one or more computer readable storage media; and
computer program instructions, the computer program instructions being stored on the one or more computer readable storage media for execution by the one or more computer processors, the computer program instructions including instructions to:
generate a saliency map corresponding to an input image, wherein the input image is an image that requires annotation;
generate a behavior saliency map, wherein the behavior saliency map is a saliency map formed from an average of a plurality of objects contained within respective bounding boxes of a plurality of sample images;
generate a historical saliency map, wherein the historical saliency map is a saliency map formed from an average of a plurality of tagged objects in the plurality of sample images;
fuse the saliency map corresponding to the input image, the behavior saliency map, and the historical saliency map to form a fused saliency map; and
generate, based on the fused saliency map, a bounding box around an object in the input image.

16. The computer system of claim 15 , wherein the instructions to fuse the saliency map corresponding to the input image, the behavior saliency map, and the historical saliency map to form the fused saliency map is further based on, at least in part, on instructions to:
assign one or more weighted values to the saliency map corresponding to the input image, the behavior saliency map, and historical saliency map, respectively.

17. The computer system of claim 16 , wherein the instructions for the one or more weighted values assigned to the saliency map corresponding to the input image, the behavior saliency map, and the historical saliency map are automatically determined based on an image attention mechanism.

18. The computer system of claim 15 , wherein the instructions to generate the saliency map corresponding to the input image further includes instructions to:
generate a hierarchical image layer; and
generate one or more saliency cues for the hierarchical image layer.

19. The computer system of claim 15 , wherein the instructions to generate the historical saliency map further includes instructions to:
select one or more sample images above a predete

Assignees

• IBM

Inventors

• Yin kun yan
• Yu Chao
• Chen Ming Jin
• Sun Teng
• Zhang Hong Bing

Classifications

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06F18/25

  Fusion techniques · CPC title

• G06V20/20 Primary

  in augmented reality scenes · CPC title