Linking contextual information to text in time dependent media

Patent metadata

Field	Value
Publication number	US-11176332-B2
Application number	US-201916535266-A
Country	US
Kind code	B2
Filing date	Aug 8, 2019
Priority date	Aug 8, 2019
Publication date	Nov 16, 2021
Grant date	Nov 16, 2021

Abstract

Time dependent media (TDM) content is received, and text features and non-text features are extracted from the TDM content. The TDM content is split into two or more documents. A document, which includes non-text features from the extracted non-text features, is selected. Non-text features in the document are compared to context patterns. When a context pattern matches a non-text feature in the document, a context element linked to the context pattern is linked to the non-text feature as well. The TDM content is modified based on the context element.

First claim

What is claimed is:

1. A system, comprising:
at least one processing component;
at least one memory component communicatively coupled to the at least one processing component, wherein the at least one processing component is configured to perform a method comprising;
receiving time dependent media (TDM) content;
extracting text features and non-text features from the TDM content;
splitting the TDM content into two or more documents;
selecting a document from the two or more documents, wherein the document includes at least one non-text feature from the non-text features;
comparing the at least one non-text feature to context patterns;
determining that a context pattern of the context patterns matches a non-text feature from the at least one non-text feature, wherein the context pattern is linked to a context element;
linking the context element to the non-text feature; and
modifying the TDM content based on the context element.

2. The system of claim 1 , wherein the method further comprises:
selecting at least one document from the two or more documents;
forming a document subset from the at least one document; and
selecting the document from the document subset.

3. The system of claim 1 , wherein the method further comprises selecting division points in the TDM content.

4. The system of claim 1 , wherein the context pattern defines a set of acoustic units.

5. The system of claim 1 , wherein the context pattern defines a facial expression.

6. The system of claim 1 , further comprising a content database.

7. The system of claim 1 , further comprising a user interface.

8. The system of claim 7 , wherein the user interface comprises a media display and a search interface.

9. A method, comprising:
receiving time dependent media (TDM) content;
extracting text features and non-text features from the TDM content;
splitting the TDM content into two or more documents;
selecting a document from the two or more documents, wherein the document includes at least one non-text feature from the non-text features;
comparing the at least one non-text feature to context patterns;
determining that a context pattern of the context patterns matches a non-text feature from the at least one non-text feature, wherein the context pattern is linked to a context element;
linking the context element to the non-text feature; and
modifying the TDM content based on the context element.

10. The method of claim 9 , wherein the splitting the TDM content comprises:
selecting division points in the TDM content; and
splitting the TDM content into the two or more documents at the division points.

11. The method of claim 9 , wherein the non-text features are mapped to the text features based on temporal positions.

12. The method of claim 9 , wherein the context pattern defines an intonation.

13. The method of claim 9 , wherein the context pattern defines a gesture.

14. The method of claim 9 , wherein the modifying comprises generating a metadata tag identifying the context element.

15. The method of claim 9 , wherein the modifying comprises displaying a portion of text in the TDM content in a color assigned to the context element.

16. A computer program product for supplementing text, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the device to perform a method, the method comprising:
receiving time dependent media (TDM) content;
extracting text features and non-text features from the TDM content;
splitting the TDM content into two or more documents;
selecting a document from the two or more documents, wherein the document includes at least one non-text feature from the extracted non-text features;
comparing the at least one non-text feature to context patterns;
determining that a context pattern of the context patterns matches a non-text feature from the at least one non-text feature, wherein the context pattern is linked to a context element;
linking the context element to the non-text feature; and
modifying the TDM content based on the context element.

17. The computer program product of claim 16 , wherein the modifying comprises annotating the TDM content.

18. The computer program product of claim 16 , wherein the modifying comprises generating synthetic speech using the context element and the text features.

19. The computer program product of claim 16 , wherein the splitting the TDM content comprises:
selecting division points in the TDM content; and
splitting the TDM content into the two or more documents at the division points.

20. The computer program product of claim 16 , wherein the non-text features are mapped to the text features based on temporal positions.

Assignees

• IBM

Inventors

• Ishikawa Shunsuke
• Tominaga Yasuyuki
• Uetsuki Hiroaki
• Ono Asako
• Hasegawa Tohru
• Yoshida Issei
• Watanabe Kenta

Classifications

• G06F40/40 Primary

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title

• G06F16/483 Primary

  using metadata automatically derived from the content · CPC title

• G06V30/413

  Classification of content, e.g. text, photographs or tables · CPC title

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82

  using neural networks · CPC title