Spatio-temporal action and actor localization

Patent metadata

Field	Value
Publication number	US-10896342-B2
Application number	US-201816189974-A
Country	US
Kind code	B2
Filing date	Nov 13, 2018
Priority date	Nov 14, 2017
Publication date	Jan 19, 2021
Grant date	Jan 19, 2021

Abstract

A method of pixel-wise localization of an actor and an action in a sequence of frames includes receiving a natural language query describing the action and the actor. The method also includes receiving the sequence of frames. The method further includes localizing the action and the actor in the sequence of frames based on the natural language query.

First claim

What is claimed is:

1. A method of pixel-wise localization of an actor and an action in a sequence of frames, comprising:
generating a first set of filters based on a natural language query describing the action and the actor;
generating a visual representation for each frame of the sequence of frames;
generating a response map for each frame of the sequence of frames based on a convolution of the first set of filters and the visual representation of each frame;
generating a second set of dynamic filters based on the natural language query and the response map;
labeling pixels in each frame of the sequence of frames based on a convolution of the second set of dynamic filters and an up-sampled visual representation of each frame; and
localizing the action and the actor in the sequence of frames based on the labeled pixels.

2. The method of claim 1 , further comprising:
up-sampling a resolution of the visual representation;
and
convolving the up-sampled visual representation with the second set of dynamic filters.

3. The method of claim 2 , further comprising repeating the up-sampling and the convolving for a set of resolutions of the visual representation.

4. The method of claim 2 , in which the first set of filters and the second set of dynamic filters are two-dimensional or three-dimensional filters.

5. The method of claim 1 , further comprising controlling an apparatus based on the localized action and actor.

6. An apparatus for pixel-wise localization of an actor and an action in a sequence of frames, the apparatus comprising:
means for generating a first set of filters based on a natural language query describing the action and the actor;
means for generating a visual representation for each frame of the sequence of frames;
means for generating a response map for each frame of the sequence of frames based on a convolution of the first set of filters and the visual representation of each frame;
means for generating a second set of dynamic filters based on the natural language query and the response map;
means for labeling pixels in each frame of the sequence of frames based on a convolution of the second set of dynamic filters and an up-sampled visual representation of each frame; and
means for localizing the action and the actor in the sequence of frames based on the labeled pixels.

7. The apparatus of claim 6 , further comprising:
means for up-sampling a resolution of the visual representation;
and
means for convolving the up-sampled visual representation with the second set of dynamic filters.

8. The apparatus of claim 7 , further comprising means for repeating up-sampling and convolving for a set of resolutions of the visual representation.

9. The apparatus of claim 7 , in which the first set of filters and the second set of dynamic filters are two-dimensional or three-dimensional filters.

10. The apparatus of claim 6 , further comprising means for controlling the apparatus based on the localized action and actor.

11. An apparatus for pixel-wise localization of an actor and an action in a sequence of frames, the apparatus comprising:
a memory; and
at least one processor coupled to the memory, the at least one processor configured:
to generate a first set of filters based on a natural language query describing the action and the actor;
to generate a visual representation for each frame of the sequence of frames;
to generate a response map for each frame of the sequence of frames based on a convolution of the first set of filters and the visual representation of each frame;
to generate a second set of dynamic filters based on the natural language query and the response map;
to label pixels in each frame of the sequence of frames based on a convolution of the second set of dynamic filters and an up-sampled visual representation of each frame; and
to localize the action and the actor in the sequence of frames based on the labeled pixels.

12. The apparatus of claim 11 , in which the at least one processor is further configured to:
up-sample a resolution of the visual representation;
and
convolve the up-sampled visual representation with the second set of dynamic filters.

13. The apparatus of claim 12 , in which the at least one processor is further configured to up-sample and convolve for a set of resolutions of the visual representation.

14. The apparatus of claim 12 , in which the first set of filters and the second set of dynamic filters are two-dimensional or three-dimensional filters.

15. The apparatus of claim 11 , in which the at least one processor is further configured to control the apparatus based on the localized action and actor.

16. A non-transitory computer-readable medium having program code recorded thereon for pixel-wise localization of an actor and an action in a sequence of frames, the program code executed by a processor and comprising:
program code to generate a first set of filters based on a natural language query describing the action and the actor;
program code to generate a visual representation for each frame of the sequence of frames;
program code to generate a response map for each frame of the sequence of frames based on a convolution of the first set of filters and the visual representation of each frame;
program code to generate a second set of dynamic filters based on the natural language query and the response map;
program code to label pixels in each frame of the sequence of frames based on a convolution of the second set of dynamic filters and an up-sampled visual representation of each frame; and
program code to localize the action and the actor in the sequence of frames based on the labeled pixels.

17. The non-transitory computer-readable medium of claim 16 , in which the program code further comprises:
program code to up-sample a resolution of the visual representation; and
program code to convolve the up-sampled visual representation with the second set of dynamic filters.

18. The non-transitory computer-readable medium of claim 17 , in which the program code further comprises program code to up-sample and program code to convolve for a set of resolutions of the visual representation.

19. The non-transitory computer-readable medium of claim 17 , in which the first set of filters and the second set of dynamic filters are two-dimensional or three-dimensional filters.

20. The non-transitory computer-readable medium of claim 16 , in which the program code further comprises program code to control an apparatus based on the localized action and actor.

Assignees

• Qualcomm Inc

Inventors

• Gavrilyuk Kirill
• Ghodrati Amir
• Li Zhenyang
• Snoek Cornelis Gerardus Maria

Classifications

• G06T7/215 Primary

  Motion-based segmentation · CPC title

• G06V40/171

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title

• G06V40/166

  using acquisition arrangements · CPC title

• G06F16/73

  Querying · CPC title

• G11B27/00

  Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel · CPC title