Detecting and predicting object events from images

Patent metadata

Field	Value
Publication number	US-11379733-B2
Application number	US-201916507094-A
Country	US
Kind code	B2
Filing date	Jul 10, 2019
Priority date	Jul 10, 2019
Publication date	Jul 5, 2022
Grant date	Jul 5, 2022

Abstract

A method for event predictions is provided. The method includes receiving input data. The method further includes identifying an object in the input data with the identified object associated with a first node in a knowledge graph. The method further includes determining a second node of a first object event with the second node related to the first node in the knowledge graph. The method further includes contextualizing the identified input object with the first object event.

First claim

What is claimed is:

1. A computer-implemented method comprising:
relating, in a knowledge graph, nodes of first objects with nodes of second objects and with nodes of object events, wherein the knowledge graph is structured into multiple hierarchical levels, at least one such level consisting of nodes for objects, and at least one such level consisting of nodes for object events;
receiving input data, the input data comprising an input image;
identifying a first object and a second object in the input image, the identified first object associated with a first node in the knowledge graph, the identified second object associated with a second node in the knowledge graph;
ranking a set of object events based on probability using the knowledge graph, the probability based on a number of edges in the knowledge graph between each node corresponding to object events and the first and second node;
predicting a first object event occurring within a time vicinity based on the ranking.

2. The method of claim 1 , wherein the ranking is further based on edge weights associated with each edge.

3. The method of claim 1 , further comprising, in response to predicting the first object event occurring within the time vicinity, triggering an action signal for an actuator.

4. The method of claim 3 , wherein the actuator a influences movement of a portion of a robot, and wherein the action signal is configured to pre-position the portion of the robot in expectation of the first object event occurring.

5. The method of claim 1 , wherein the identifying is performed by pattern detection or classification.

6. The method of claim 1 , wherein the levels are differentiated by their respective value of conceptual abstraction.

7. The method of claim 1 , wherein the input data further comprises sound data, the method further comprising:
identifying a third object in the sound data, wherein the ranking is further based on the number of edges in the knowledge graph between each node corresponding to the object events and a third node associated with the third object.

8. A system comprising:
a memory; and
a processor in communication with the memory, the system configured to perform a method comprising:
relating, in a knowledge graph, nodes of first objects with nodes of second objects and with nodes of object events, wherein the knowledge graph is structured into multiple hierarchical levels, at least one such level consisting of nodes for objects, and at least one such level consisting of nodes for object events;
receiving input data, the input data comprising an input image;
identifying a first object and a second object in the input image, the identified first object associated with a first node in the knowledge graph, the identified second object associated with a second node in the knowledge graph;
ranking a set of object events based on probability using the knowledge graph, the probability based on a number of edges in the knowledge graph between each node corresponding to object events and the first and second node;
predicting a first object event occurring within a time vicinity based on the ranking.

9. The system of claim 8 , wherein the ranking is further based on edge weights associated with each edge.

10. The system of claim 8 , wherein the method further comprises triggering an action signal for an actuator based on the predicting the first object event occurring within the time vicinity.

11. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by one or more computing systems to cause the one or more computing systems to perform a method comprising:
relating, in a knowledge graph, nodes of first objects with nodes of second objects and with nodes of object events, wherein the knowledge graph is structured into multiple hierarchical levels, at least one such level consisting of nodes for objects, and at least one such level consisting of nodes for object events;
receiving input data, the input data comprising an input image;
identifying a first object and a second object in the input image, the identified first object associated with a first node in the knowledge graph, the identified second object associated with a second node in the knowledge graph;
ranking a set of object events based on probability using the knowledge graph, the probability based on a number of edges in the knowledge graph between each node corresponding to object events and the first and second node;
predicting a first object event occurring within a time vicinity based on the ranking.

12. The computer program product of claim 11 , wherein the ranking is further based on edge weights associated with each edge.

Assignees

• IBM

Inventors

• Giovannini Andrea
• Flöther Frederik Frank
• Graf Florian
• Ravizza Stefan
• Rueger Erik

Classifications

• G06V20/42

  of sport video content · CPC title

• G06F16/583 Primary

  using metadata automatically derived from the content · CPC title

• G06N5/02 Primary

  Knowledge representation; Symbolic representation · CPC title

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title