Multimodal input processing

Patent metadata

Field	Value
Publication number	US-9530102-B2
Application number	US-201514623985-A
Country	US
Kind code	B2
Filing date	Feb 17, 2015
Priority date	Feb 17, 2015
Publication date	Dec 27, 2016
Grant date	Dec 27, 2016

Abstract

Exemplary embodiments described herein provide methods, mediums, and systems for recognizing a formalism type present in an input and generating an output that is consistent with the formalism type. In order to generate an output that is consistent with the formalism type, exemplary embodiments analyze elements of the input together. Such a holistic analysis determines or uses a likelihood that the different elements coexist together in a given formalism type. Based on this holistic analysis, an appropriate formalism type that is consistent with the coexistence of the elements may be selected. After the appropriate formalism type is selected, the input may be processed to generate an output consistent with the formalism type. The output may be computer-based representations of the input defined according to a program associated with the identified formalism type, and/or the input may be translated or transformed into another representation.

First claim

The invention claimed is:

1. A non-transitory computer readable medium storing instructions, the instructions comprising:
one or more instructions that, when executed, cause a processor to:
receive an input, the input comprising a plurality of input elements;
parse the input to identify:
a first element from among the plurality of input elements, the first element represented according to a formalism type that identifies having a syntax for organizing elements of the formalism type, and
a second element from among the plurality of input elements, the second element represented according to the formalism type;

access a library comprising entries for different formalism types, the formalism type included in the different formalism types;
determine a likelihood that the first element and the second element coexist in a selected formalism type from among the different types of formalisms in the library, and
select, based on the determining, the selected formalism type that is consistent with a coexistence of the first element and the second element;
wherein the selected formalism type is one of the following: an equation, a state diagram, a plot, a chart, a table, a Unified Modeling Language (UML) diagram, a chemical formula, a block diagram model, a type of diagram, a multi-body modeling formalism, a computer-aided design formalism, a physical modeling formalism or source code,
generate an output, the output:
comprising the first element and the second element, and
at least partly represented in a format corresponding to the selected formalism type; and
wherein if the selected formalism type is an equation, the format is a solution to the equation or a source code function representing the equation;
wherein if the selected formalism type is a state diagram, the format is a state chart model representing the state diagram;
wherein if the selected formalism type is a plot, the format is a source code command that when executed generates the plot or a table of data represented in the plot;
wherein if the selected formalism type is a table, the format is a plot or a chart representing the table;
wherein if the selected formalism type is a chart, the format is a table of data represented in the chart;
wherein if the selected formalism type is a UML diagram, the format is a source code class representing the UML diagram;
wherein if the selected formalism type is a chemical formula, the format is a name of a chemical represented by the chemical formula;
wherein if the selected formalism type is a block diagram model, the format is a source code representation of the block diagram model; and
wherein if the selected formalism type represents source code, the format is a block diagram model.

2. The medium of claim 1 , wherein the plurality of input elements of the input include a first set of elements represented according to a first formalism type and a second set of elements represented according to a second formalism type distinct from the first formalism type.

3. The medium of claim 1 , wherein the output comprises output elements corresponding to a plurality of distinct formalisms such that the output is partly represented in the format corresponding to the selected formalism type, and is further represented in a second format corresponding to a second selected formalism type that is distinct from the selected formalism type.

4. The medium of claim 1 , wherein different formalism types are processed by different types of processing engines or environments.

5. The medium of claim 1 , wherein different formalism types have different alphabets, syntaxes, or semantics.

6. The medium of claim 1 , wherein the input comprises one or more of:
an image;
video;
audio; and
gesture data.

7. The medium of claim 1 , wherein the library comprises a table of probabilities that a type of element is found in each of the formalism types in the library.

8. The medium of claim 7 , wherein the probabilities are dynamically updated based on user-feedback about a classification of the input into a formalism type.

9. The medium of claim 7 , wherein the probabilities are dynamically updated based on historical classifications of formalism types.

10. The medium of claim 7 , wherein analyzing the first element and the second element comprises:
determining a first probability, the first probability representing a probability that the first element belongs to the selected formalism type;
determining a second probability, the second probability representing a probability that the second element belongs to the selected formalism type; and
adding the first probability to the second probability.

11. The medium of claim 1 , wherein the second element is identified, at least in part, based on a predicted formalism type that is determined based on an analysis of the first element.

12. The medium of claim 1 , wherein generating the output comprises applying a formalism-specific rule to the first element or the second element, the formalism-specific rule being particular to the selected formalism type.

13. The medium of claim 1 , wherein generating the output comprises processing the first element and the second element using a formalism-specific procedure.

14. The medium of claim 1 , further comprising displaying help information specific to the selected formalism type, the first element, or the second element.

15. A method comprising:
receiving, using a processor of a computing device, an input, the input comprising a plurality of input elements;
parsing the input, the parsing:
performed using the processor,
identifying a first element from among the plurality of input elements, and
identifying a second element from among the plurality of input elements,
wherein the first element and the second element are represented according to a formalism type defining a syntax for organizing elements of the formalism type;

access a library comprising entries for different formalism types;
evaluating a likelihood that the first element and the second element coexist together in a selected formalism type from among the different formalism types in the library, and
selecting, based on the evaluating, the selected formalism type that is consistent with a coexistence of the first element and the second element;
wherein the selected formalism type is one of the following: an equation, a state diagram, a plot, a chart, a table, a Unified Modeling Language (UML) diagram, a chemical formula, a block diagram model, a type of diagram, a multi-body modeling formalism, a computer aided design formalism, a physical modeling-formalism or source code,
generating, using the processor, an output, the output:
representing a translation of at least part of the input into an output type that is associated with the selected formalism type; and
wherein if the selected formalism type is an equation, the translation is a solution to the equation or a source code function representing the equation;
wherein if the selected formalism type is a state diagram, the translation is a state chart model representing the state diagram;
wherein if the selected formalism type is a plot, the translation is a source code command that when executed generates the plot or a table of data represented in the plot;
wherein if the selected formalism type is a table, the translation is a plot or a chart representing the table;
wherein if the selected formalism type is a chart, the translation is a table of data represented in the chart;
wherein if the selected formalism type is a UML diagram, the translation is a source code class representing the UML diag

Assignees

• Mathworks Inc

Inventors

• Mosterman Pieter J
• Parupati Srikanth
• Desai Ankit A
• Uthama Ashish
• Quinn Michael J

Classifications

• G06N7/01 Primary

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N7/005 Primary

  Physics · mapped topic

• G06N5/02

  Knowledge representation; Symbolic representation · CPC title

• G06V30/422 Primary

  Technical drawings; Geographical maps · CPC title

• G06V30/268

  Lexical context · CPC title