Situational awareness by fusing multi-modal data with semantic model

Patent metadata

Field	Value
Publication number	US-2022019742-A1
Application number	US-202016933964-A
Country	US
Kind code	A1
Filing date	Jul 20, 2020
Priority date	Jul 20, 2020
Publication date	Jan 20, 2022
Grant date	—

Abstract

A method is provided for creating a semantic model for submitting search queries thereto. The method includes an act of receiving data from one or more input sources in an entity and relationship capture service of a situational awareness engine. The method further includes an act of extracting entities and relationships between the entities in two or more extraction services, where the two or more extraction services include at least two of a table-to-graph service, an event-to-graph service, a sensor-to-graph service, a text-to-graph service, and an image-to-graph service. The method includes an act of generating a semantic model based on fusion and labeling the extracted data provided by the at least two extraction services, where the semantic model can receive a search query and respond to the search query based on the generated semantic model.

First claim

What is claimed is:

1 . A computer implemented method, comprising:
receiving data from one or more input sources in an entity and relationship capture service of a situational awareness engine; extracting entities and relationships between the entities in two or more extraction services, the two or more extraction services including at least two of a table-to-graph service, an event-to-graph service, a sensor-to-graph service, a text-to-graph service, and an image-to-graph service; generating a semantic model based on fusion and labeling of extracted data provided by the at least two extraction services; receiving a search query; and responding to the search query based on the generated semantic model.

2 . The computer implemented method of claim 1 , wherein the semantic model is a knowledge graph with nodes representing entities and edges representing relationships between the entities.

3 . The computer implemented method of claim 1 , wherein the data comprises two or more disparate data types.

4 . The computer implemented method of claim 1 , further comprising receiving domain input on entities and relationships from a user via a user interface of a user computing device.

5 . The computer implemented method of claim 1 , further comprising providing a portion of the extracted data to a user via a user interface of a user computing device for editing.

6 . The computer implemented method of claim 5 , further comprising providing a labeling service for automatically reviewing the extracted data to identify disconnected components of data and providing details of the disconnected component to the user.

7 . The computer implemented method of claim 1 , further comprising reconciling entities across the one or more input sources.

8 . The computer implemented method of claim 7 , further comprising merging, with a fusion service of the situational awareness engine, two entities extracted from two of the one or more input sources upon determining that the two entities are the same.

9 . The computer implemented method of claim 7 , further comprising determining a relationship between two related entities from two of the one or more input sources.

10 . The computer implemented method of claim 1 , wherein the search query of the semantic model is for problem diagnosis upon the situation awareness engine detecting a problem.

11 . A computerized situational awareness system comprising:
a situational awareness engine having two or more extraction services, where the two or more extraction services include at least two of a table-to-graph service, an event-to-graph service, a sensor-to-graph service, a text-to-graph service and an image-to-graph service; a user interface permitting a user to review data extracted by the two or more extraction services; an entity and relationship capture engine of the situational awareness engine receiving data from one or more input sources; and a semantic model generated by the situational awareness engine based on fusion and labeling of data extracted by the two or more extraction services.

12 . The computerized situational awareness system of claim 11 , wherein the semantic model is a knowledge graph with nodes representing entities and edges representing relationships between the entities.

13 . The computerized situational awareness system of claim 11 , wherein the one or more input sources provides two or more disparate data types.

14 . A non-transitory computer readable storage medium tangibly embodying a computer readable program code having computer readable instructions that, when executed, causes a computer device to carry out a method of improving computing efficiency of a computing device for problem solving and reasoning, the method comprising:
receiving data from one or more input sources into a situational awareness engine; capturing the data in an entity and relationship capture service; providing the data to one or more extraction services, the situational awareness engine including at least two extraction services selected from the group consisting of a table-to-graph service, an event-to-graph service, a sensor-to-graph service, a text-to-graph service and an image-to-graph service; and generating a semantic model based on fusion and labeling of extracted data provided by the at least two extraction services.

15 . The non-transitory computer readable storage medium of claim 14 , wherein the semantic model is a knowledge graph with nodes representing entities and edges representing relationships between the entities.

16 . The non-transitory computer readable storage medium of claim 14 , wherein the execution of the code by the processor further configures the computing device to perform an act comprising providing a portion of the extracted data to a user via a user interface for editing.

17 . The non-transitory computer readable storage medium of claim 16 , wherein the execution of the code by the processor further configures the computing device to perform an act comprising providing a labeling service for automatically reviewing the extracted data to identify disconnected components of data and providing details of the disconnected component to the user.

18 . The non-transitory computer readable storage medium of claim 15 , wherein the execution of the code by the processor further configures the computing device to perform an act comprising reconciliating entities across the one or more input sources.

19 . The non-transitory computer readable storage medium of claim 18 , wherein the execution of the code by the processor further configures the computing device to perform an act comprising merging, with a fusion service of the situational awareness engine, two entities extracted from two of the one or more input sources when it is determined the two entities are the same.

20 . The non-transitory computer readable storage medium of claim 14 , wherein the execution of the code by the processor further configures the computing device to perform an act comprising performing a search of the semantic model for problem diagnosis.

Assignees

• IBM

Inventors

• Bhamidipaty Anuradha
• Khabiri Elham
• Lin Shuxin
• Kephart Jeffrey Owen
• Li Yingjie
• Agrawal Bhavna

Classifications

• G06N5/022

  Knowledge engineering; Knowledge acquisition · CPC title

• G06F40/295 Primary

  Named entity recognition · CPC title

• G06F40/30 Primary

  Semantic analysis · CPC title

• G06N5/02

  Knowledge representation; Symbolic representation · CPC title