Multimodal method for detecting video, multimodal video detecting system and non-transitory computer readable medium

Patent metadata

Field	Value
Publication number	US-12014546-B2
Application number	US-202117456375-A
Country	US
Kind code	B2
Filing date	Nov 23, 2021
Priority date	Nov 3, 2021
Publication date	Jun 18, 2024
Grant date	Jun 18, 2024

Abstract

A multimodal method for detecting video includes following step of: receiving a message to be detected to obtain a multimodal association result, which message to be detected corresponds to a video to be detected; generating a plurality of detecting conditions according to multimodal association result; searching a plurality of videos in a video detection database to obtain a target video in videos according to detecting conditions, which each of videos includes a plurality of video paragraphs respectively, which each of video paragraphs includes a piece of multimodal related data respectively; comparing detecting conditions and piece of multimodal related data of video paragraphs to obtain a matching video paragraph and using video corresponding to matching video paragraph as the target video; and outputting the target video and the video to be detected to a display device for display.

First claim

What is claimed is:

1. A multimodal method for detecting a video, comprising:
receiving a message to be detected so as to obtain a multimodal association result by a processor, wherein the message to be detected corresponds to a video to be detected, wherein the message to be detected further comprises a natural language text paragraph corresponding to the video to be detected, wherein receiving the message to be detected so as to obtain the multimodal association result by the processor comprises:
receiving the video to be detected and the natural language text paragraph corresponding to the video to be detected to obtain the multimodal association result by the processor;
generating a plurality of detecting conditions according to the multimodal association result by the processor;
searching a plurality of videos in a video detection database so as to obtain a target video in the plurality of videos according to the plurality of detecting conditions by the processor, wherein each of the plurality of videos comprises a plurality of video paragraphs respectively, wherein each of the plurality of video paragraphs comprises a piece of multimodal related data respectively;
comparing the plurality of detecting conditions and the piece of multimodal related data of the plurality of video paragraphs so as to obtain a matching video paragraph, and using a video corresponding to the matching video paragraph as the target video by the processor; and
outputting the target video and the video to be detected to a display device for display by the processor.

2. The multimodal method for detecting the video of claim 1 , wherein the piece of multimodal related data of the plurality of video paragraphs is established by following steps:
downloading the plurality of videos from a plurality of video databases so as to store the plurality of videos into the video detection database by a server;
identifying a plurality of modal characteristics of the plurality of video paragraphs of each of the plurality of videos by the server;
generating the piece of multimodal related data corresponding to the plurality of video paragraphs according to the plurality of modal characteristics by the server; and
marking the plurality of video paragraphs of each of the plurality of videos according to the piece of multimodal related data of the plurality of video paragraphs so as to store the plurality of videos in the video detection database by the server respectively.

3. The multimodal method for detecting a video of claim 2 , wherein a modal type of the plurality of modal characteristics comprises one of a semantic recognition modality, a natural language recognition modality, a face recognition modality, a building recognition modality, an object recognition modality, a behavior recognition modality, a number recognition modality, and a color recognition modality.

4. The multimodal method for detecting a video of claim 1 , wherein the plurality of detecting conditions are a piece of relevance descriptive data corresponding to the video to be detected, wherein the piece of relevance descriptive data comprises one of a behavior of a person, a behavior of an animal, an action of a person, an action of an animal, an facial expression of a person, an facial expression of an animal, an emotion of a person, an emotion of an animal, a relative position between a person and a place/building, a relative position between an animal and a place/building, a relative relationship between a person, other people, and things, and an event.

5. The multimodal method for detecting the video of claim 1 , wherein receiving the message to be detected so as to obtain the multimodal association result by the processor further comprises:
receiving the video to be detected by the processor;
identifying a plurality of modal characteristics of a plurality of video paragraphs of the video to be detected by the processor; and
obtaining the multimodal association result according to the plurality of modal characteristics and the natural language text paragraph by the processor.

6. The multimodal method for detecting the video of claim 5 , wherein a modal type of the plurality of modal characteristics comprises one of a semantic recognition modality, a natural language recognition modality, a face recognition modality, a building recognition modality, an object recognition modality, a behavior recognition modality, a number recognition modality, and a color recognition modality.

7. The multimodal method for detecting the video of claim 1 , further comprising:
comparing the multimodal association result and the piece of multimodal related data of one of the video paragraphs of the target video so as to obtain a target video paragraph of the target video by the processor; and
displaying the target video paragraph and the video to be detected to the display device by the processor.

8. A multimodal video detecting system, comprising:
a memory, configured to store a plurality of instructions; and
a processor, coupled to the memory, wherein the processor is configured to read the plurality of instructions to perform following steps:
receiving a message to be detected so as to obtain a multimodal association result, wherein the message to be detected corresponds to a video to be detected;
generating a plurality of detecting conditions according to the multimodal association result;
searching a plurality of videos in a video detection database so as to obtain a target video in the plurality of videos according to the plurality of detecting conditions, wherein each of the plurality of videos comprises a plurality of video paragraphs respectively, wherein each of the plurality of video paragraphs comprises a piece of multimodal related data respectively;
comparing the plurality of detecting conditions and the piece of multimodal related data of the plurality of video paragraphs so as to obtain a matching video paragraph and using a video corresponding to the matching video paragraph as the target video; and
outputting the target video and the video to be detected to a display device for display, wherein the message to be detected further comprises a natural language text paragraph corresponding to the video to be detected, wherein the processor is further configured to receive the video to be detected and the natural language text paragraph corresponding to the video to be detected to obtain the multimodal association result.

9. The multimodal video detecting system of claim 8 , wherein the piece of multimodal related data of the plurality of video paragraphs is established by following steps:
downloading the plurality of videos from a plurality of video databases so as to store the plurality of videos into the video detection database by a server;
identifying a plurality of modal characteristics of the plurality of video paragraphs of each of the plurality of videos by the server;
generating the piece of multimodal related data corresponding to the plurality of video paragraphs according to the plurality of modal characteristics by the server; and
marking the plurality of video paragraphs of each of the plurality of videos according to the piece of multimodal related data of the plurality of video paragraphs so as to store the plurality of videos in the video detection database by the server respectively.

10. The multimodal video detecting system of claim 9 , wherein a modal type of the plurality of modal characteristics comprises one of a semantic recognition modality, a natural language recognition modality, a face recognition modality, a building recognition modality, an object recognition modality, a behavior recognition modality, a number recognition modality, an

Assignees

• Inst Information Ind

Inventors

• Huang Chu-Chun
• Tsao Yen-Heng
• Chen Ping-I

Classifications

• G06V40/164

  using holistic features · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06V40/176

  Dynamic expression · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06F16/743

  a collection of video files or sequences · CPC title