Adaptive selection of data modalities for efficient video recognition

Patent metadata

Field	Value
Publication number	US-12249147-B2
Application number	US-202117199307-A
Country	US
Kind code	B2
Filing date	Mar 11, 2021
Priority date	Mar 11, 2021
Publication date	Mar 11, 2025
Grant date	Mar 11, 2025

Abstract

One embodiment of the invention provides a method for video recognition. The method comprises receiving an input video comprising a sequence of video segments over a plurality of data modalities. The method further comprises, for a video segment of the sequence, selecting one or more data modalities based on data representing the video segment. Each data modality selected is optimal for video recognition of the video segment. The method further comprises, for each data modality selected, providing at least one data input representing the video segment over the data modality selected to a machine learning model corresponding to the data modality selected, and generating a first type of prediction representative of the video segment via the machine learning model. The method further comprises determining a second type of prediction representative of the entire input video by aggregating all first type of predictions generated.

First claim

What is claimed is:

1. A method for video recognition, comprising: receiving an input video comprising a sequence of video segments over a plurality of data modalities, wherein each segment comprising two or more frames;
for at least one video segment of the sequence of video segments, adaptively selecting a subset of data modalities of the plurality of data modalities based on data representing the video segment, wherein each data modality selected is optimal for video recognition of the at least one video segment, and wherein each data modality of the plurality of data modalities that is not selected is redundant for the video recognition of the at least one video segment, wherein the plurality of data modalities are selected from a group comprising a RGB modality, an optical flow modality, and an audio modality;
for each data modality selected, providing at least one data input representing the at least one video segment over the data modality selected to a machine learning model corresponding to the data modality selected, and generating a first type of prediction representative of the at least one video segment via the machine learning model; and
determining a second type of prediction representative of the input video as a whole by aggregating all first type of predictions generated, wherein the second type of prediction is indicative of an object or an activity captured in the input video.

2. The method of claim 1 , wherein the data representing the video segment comprise at least one of one or more RGB frames, one or more RGB difference frames, and one or more audio frames.

3. The method of claim 1 , wherein each data modality of the plurality of data modalities has a corresponding machine learning model that is jointly trained with one or more other machine learning models corresponding to one or more other data modalities of the plurality of data modalities.

4. The method of claim 3 , wherein each machine learning model corresponding to each data modality of the plurality of data modalities comprises a sub-network.

5. The method of claim 1 , wherein the one or more data modalities selected provide an optimal trade-off between video recognition accuracy and computational efficiency.

6. The method of claim 1 , further comprising:
extracting, via a joint feature extractor, a joint feature from the data inputs representing the video segment over the plurality of data modalities;
computing, via a long short-term memory (LSTM), hidden states for the video segment based in part on the joint feature extracted; and
for each data modality of the plurality of data modalities:
estimating a corresponding policy distribution based on the hidden states for the video segment; and
applying a Gumbel-Softmax operation to the corresponding policy distribution to sample a corresponding binary decision indicative of whether to select the data modality for the video recognition of the video segment.

7. A system for video recognition, comprising: at least one processor; and a non-transitory processor-readable memory device storing instructions that when executed by the at least one processor causes the at least one processor to perform operations including:
receiving an input video comprising a sequence of video segments over a plurality of data modalities, wherein each segment comprising two or more frames;
for at least one video segment of the sequence of video segments, adaptively selecting a subset of data modalities of the plurality of data modalities based on data representing the video segment, wherein each data modality selected is optimal for video recognition of the at least one video segment, and wherein each data modality of the plurality of data modalities that is not selected is redundant for the video recognition of the at least one video segment, wherein the plurality of data modalities are selected from a group comprising a RGB modality, an optical flow modality, and an audio modality;
for each data modality selected, providing at least one data input representing the at least one video segment over the data modality selected to a machine learning model corresponding to the data modality selected, and generating a first type of prediction representative of the at least one video segment via the machine learning model; and
determining a second type of prediction representative of the input video as a whole by aggregating all first type of predictions generated, wherein the second type of prediction is indicative of an object or an activity captured in the input video.

8. The system of claim 7 , wherein the data representing the video segment comprise at least one of one or more RGB frames, one or more RGB difference frames, and one or more audio frames.

9. The system of claim 7 , wherein each data modality of the plurality of data modalities has a corresponding machine learning model that is jointly trained with one or more other machine learning models corresponding to one or more other data modalities of the plurality of data modalities.

10. The system of claim 9 , wherein each machine learning model corresponding to each data modality of the plurality of data modalities comprises a sub-network.

11. The system of claim 7 , wherein the one or more data modalities selected provide an optimal trade-off between video recognition accuracy and computational efficiency.

12. The system of claim 7 , wherein the instructions further include:
extracting, via a joint feature extractor, a joint feature from the data inputs representing the video segment over the plurality of data modalities;
computing, via a long short-term memory (LSTM), hidden states for the video segment based in part on the joint feature extracted; and
for each data modality of the plurality of data modalities:
estimating a corresponding policy distribution based on the hidden states for the video segment; and
applying a Gumbel-Softmax operation to the corresponding policy distribution to sample a corresponding binary decision indicative of whether to select the data modality for the video recognition of the video segment.

13. A non-transitory computer program product for video recognition, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to: receive an input video comprising a sequence of video segments over a plurality of data modalities, wherein each segment comprising two or more frames;
for at least one video segment of the sequence of video segments, adaptively select a subset of data modalities of the plurality of data modalities based on data representing the video segment, wherein each data modality selected is optimal for video recognition of the at least one video segment, and wherein each data modality of the plurality of data modalities that is not selected is redundant for the video recognition of the at least one video segment, wherein the plurality of data modalities are selected from a group comprising a RGB modality, an optical flow modality, and an audio modality;
for each data modality selected, provide at least one data input representing the at least one video segment over the data modality selected to a machine learning model corresponding to the data modality selected, and generating a first type of prediction representative of the at least one video segment via the machine learning mode; and
determine a second type of prediction representative of the input video as a whole by aggregating all first type of predictions generated, wherein the second type of prediction is indicative of an object or an activity capture

Assignees

• IBM

Inventors

• Panda Rameswar
• Chen Richard
• Fan Quanfu
• Feris Rogerio Schmidt

Classifications

• G06F18/256

  of results relating to different input data, e.g. multimodal recognition · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V20/49

  Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes · CPC title

• G06N20/00

  Machine learning · CPC title

• G06V10/88

  Image or video recognition using optical means, e.g. reference filters, holographic masks, frequency domain filters or spatial domain filters · CPC title