Temporal saliency map

Patent metadata

Field	Value
Publication number	US-10121256-B2
Application number	US-201415326614-A
Country	US
Kind code	B2
Filing date	Aug 28, 2014
Priority date	Aug 28, 2014
Publication date	Nov 6, 2018
Grant date	Nov 6, 2018

Abstract

A method of generating a temporal saliency map is disclosed. In a particular embodiment, the method includes receiving an object bounding box from an object tracker. The method includes cropping a video frame based at least in part on the object bounding box to generate a cropped image. The method further includes performing spatial dual segmentation on the cropped image to generate an initial mask and performing temporal mask refinement on the initial mask to generate a refined mask. The method also includes generating a temporal saliency map based at least in part on the refined mask.

First claim

What is claimed is:

1. A method comprising:
receiving information associated with an object bounding box from an object tracker;
cropping a video frame based at least in part on the object bounding box to generate a cropped image;
performing, at a processor, spatial dual-layer segmentation on the cropped image to generate an initial mask;
performing, at the processor, temporal mask refinement on the initial mask to generate a refined mask; and
generating a temporal saliency map based at least in part on the refined mask.

2. The method of claim 1 , further comprising providing the temporal saliency map to a special effect application.

3. The method of claim 2 , wherein the special effect application is a Bokeh effect application.

4. The method of claim 2 , wherein the special effect application is a black and white effect application.

5. The method of claim 2 , wherein the special effect application is a lens effect application.

6. The method of claim 1 , further comprising generating a binary mask based on the temporal saliency map.

7. The method of claim 6 , further comprising applying the binary mask to the video frame for a special effect application.

8. The method of claim 1 , wherein performing the spatial dual-layer segmentation includes:
performing region grow segmentation on a first layer of an image based on a first set of seed points; and
performing region grow segmentation on a second layer of the image based on a second set of seed points.

9. The method of claim 8 , wherein the first set of seed points includes a first plurality of points associated with a first boundary region that is expanded in size with respect to the object bounding box.

10. The method of claim 9 , wherein the second set of seed points includes a second plurality of points associated with a second boundary region that is reduced in size with respect to the object bounding box.

11. The method of claim 10 , further comprising fusing results of the region grow segmentation on the first layer of the image with results of the region grow segmentation on the second layer of the image to generate the initial mask.

12. An apparatus comprising:
a processor;
an object tracker component executable by the processor to generate information associated with an object bounding box;
a frame cropping component executable by the processor to crop a video frame based at least in part on the information associated with the object bounding box to generate a cropped image;
a segmentation component executable by the processor to perform spatial multiple-layer segmentation on the cropped image to generate an initial mask;
a mask refinement component executable by the processor to perform temporal mask refinement on the initial mask to generate a refined mask; and
a temporal saliency map generation component executable by the processor to generate a temporal saliency map based at least in part on the refined mask.

13. The apparatus of claim 12 , wherein the spatial multiple-layer segmentation includes spatial dual-layer segmentation.

14. The apparatus of claim 12 , wherein the mask refinement component is executable by the processor to align the initial mask with one or more masks associated with one or more second video frames that precede the video frame in a sequence of video frames.

15. The apparatus of claim 14 , wherein the mask refinement component is executable by the processor to align the initial mask with the one or more masks associated with the one or more second video frames by removing one or more pixels that represent outliers for a particular video frame.

16. The apparatus of claim 15 , wherein the one or more pixels appear in the particular video frame but do not appear in at least one of the one or more second video frames.

17. The apparatus of claim 12 , further comprising a camera to capture the video frame.

18. A method comprising:
receiving information associated with an object bounding box from an object tracker;
performing, at a processor, spatial dual-layer segmentation on a portion of a video frame to generate an initial mask, wherein a boundary of the portion of the video frame is determined based at least in part on the information associated with the object bounding box;
performing, at the processor, temporal mask refinement on the initial mask to generate a refined mask;
generating a temporal saliency map based at least in part on the refined mask; and
providing the temporal saliency map as feedback to the object tracker.

19. The method of claim 18 , wherein the temporal saliency map is provided to a normalized cross-correlation (NCC) verification component that is associated with a detection component of the object tracker.

20. The method of claim 19 , wherein the NCC verification component is configured to apply the temporal saliency map to an image in order to separate a foreground portion of the image that includes an object of interest from a background portion of the image.

21. The method of claim 20 , wherein the NCC verification component is configured to determine a subset of pixels within the object bounding box based on the temporal saliency map, wherein the subset of pixels are to be used by the NCC verification component for one or more comparison operations.

22. The method of claim 21 , wherein the subset of pixels represents the foreground portion of the image that includes the object of interest.

23. The method of claim 22 , wherein a second subset of pixels representing the background portion of the image is not used by the NCC verification component for the one or more comparison operations.

24. The method of claim 19 , wherein the NCC verification component utilizes an object appearance model for object verification, and wherein the object appearance model is updated based at least in part on an output of the NCC verification component associated with the temporal saliency map.

25. An apparatus comprising:
a processor;
an object tracker component executable by the processor to generate information associated with an object bounding box;
a segmentation component executable by the processor to perform spatial dual-layer segmentation on a portion of a video frame to generate an initial mask, wherein a boundary of the portion of the video frame is determined based at least in part on the information associated with the object bounding box;
a mask refinement component executable by the processor to perform temporal mask refinement on the initial mask to generate a refined mask; and
a temporal saliency map generation component executable by the processor to:
generate a temporal saliency map based at least in part on the refined mask; and
provide the temporal saliency map as feedback to the object tracker component.

26. The apparatus of claim 25 , wherein the temporal saliency map is further generated based on a saliency map associated with a second video frame that precedes the video frame in a sequence of video frames.

27. The apparatus of claim 26 , wherein a first weighting factor is applied to the saliency map associated with the second video frame.

28. The apparatus of claim 27 , wherein a second weighting factor is applied to the refined mask.

29. The apparatus of claim 28 , wherein the second weighting factor is determined based on the first weighting factor.

30. The apparatus o

Assignees

• Qualcomm Inc
• Zhong Xin
• Sun Yu
• Gao Dashan

Inventors

• Zhong Xin
• Sun Yu
• Gao Dashan

Classifications

• G06T2207/20156

  Automatic seed setting · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T2207/20164

  Salient point detection; Corner detection · CPC title

• G06T7/248 Primary

  involving reference images or patches · CPC title

• G06T7/187

  involving region growing; involving region merging; involving connected component labelling · CPC title