Multimodal procedural guidance content creation and conversion methods and systems

Patent metadata

Field	Value
Publication number	US-12499630-B2
Application number	US-202318203495-A
Country	US
Kind code	B2
Filing date	May 30, 2023
Priority date	Apr 20, 2022
Publication date	Dec 16, 2025
Grant date	Dec 16, 2025

Abstract

This disclosure and exemplary embodiments described herein provide methods and systems for multimodal procedural guidance content creation and conversion, however, it is to be understood that the scope of this disclosure is not limited to such application. One of the implementations described herein is related to the generation of content/instruction set that can be viewed in different modalities, including but not limited to mixed reality, VR, and audio text, however it is to be understood that the scope of this disclosure is not limited to such application.

First claim

What is claimed is:

1 . A method for parallel content authoring conversion, the method comprising:
providing, by a memory storage device, a legacy document for conversion; identifying one or more procedure steps of a source procedure from one or more source information of the legacy document; parsing the identified one or more procedure steps into one or more components; categorizing the parsed one or more components; generating one or more workflow images or one or more workflow videos from one or more source images or from known information of the one or more procedure steps; storing, by the memory storage device, the categorized one or more components into a workflow data structure as one or more converted workflow steps; associating the one or more converted workflow steps with a portion of a 3D model of equipment within a mixed reality environment; and enabling, through the mixed reality environment, access and editing of the 3D model of equipment; wherein the steps are identified by of one or more methods of identifications consisting of: a user manually selecting the steps, programmatically, observed interactions, and machine learning algorithms.

2 . The method of claim 1 , wherein the legacy document is selected from the group consisting of: video, text, voice recordings, and augmented reality.

3 . The method of claim 1 , wherein the one or more components are parsed by one or more methods of parsing consisting of artificial intelligence classification or advanced human tool sets.

4 . The method of claim 1 , further comprising leveraging artificial intelligence-based technology by generating 3D scene information from prompts or extracting visual information from multimedia sources.

5 . The method of claim 1 , wherein the one or more components is selected from the group of characteristics consisting of: verbs, objects, tools used, and reference images.

6 . The method of claim 1 , wherein the legacy document is a plurality of unstructured data.

7 . The method of claim 1 , wherein the legacy document is a plurality of interactive modality-derived information.

8 . The method of claim 1 , wherein each of the categorized one or more components comprises a maintenance action, a maintenance tool, and maintenance instructions.

9 . A system for parallel content authoring conversion, the system comprising a non-transitory computer readable medium storing instruction for execution by a processor, wherein the instructions comprise instructions to:
provide, by a memory storage device, a legacy document for conversion; identify one or more steps of a source procedure from one or more source information of the legacy document; parse the identified one or more procedure steps into one or more components; categorize the parsed one or more components; generate one or more workflow images or one or more workflow videos from one or more source images or from known information of the one or more procedure steps; store, by the memory storage device, the categorized one or more components into a workflow data structure as one or more converted workflow steps; associate the one or more converted workflow steps with a portion of a 3D model of equipment within a mixed reality environment; and enable, through the mixed reality environment, access and editing of the 3D model of equipment; wherein the steps are identified by of one or more methods of identifications consisting of: a user manually selecting the steps, programmatically, observed interactions, and machine learning algorithms.

10 . The method of claim 9 , wherein the legacy document is selected from the group consisting of: video, text, voice recordings, and augmented reality.

11 . The method of claim 9 , wherein the one or more components are parsed by one or more methods of parsing consisting of artificial intelligence classification or advanced human tool sets.

12 . The method of claim 9 , further comprising: leverage Artificial intelligence-based technology by generating 3D scene information from prompts or extracting visual information from multimedia sources.

13 . The method of claim 9 , wherein the one or more components is selected from the group of characteristics consisting of: verbs, objects, tools used, and reference images.

14 . The method of claim 9 , wherein the legacy document is a plurality of unstructured data.

15 . The method of claim 9 , wherein the legacy document is a plurality of interactive modality-derived information.

16 . The method of claim 9 , wherein each of the categorized one or more components comprises a maintenance action, a maintenance tool, and maintenance instructions.

17 . A parallel content authoring conversion device comprising:
a processor, a memory device, and an augmented reality system configured to perform a method comprising: providing, by a memory storage device, a legacy document for conversion; identifying one or more steps of a source procedure from one or more source information of the legacy document; parsing the identified one or more procedure steps into one or more components; categorizing the parsed one or more components; generating one or more workflow images or one or more workflow videos from one or more source images or from known information of the one or more procedure steps; storing, by the memory storage device, the categorized one or more components into a workflow data structure as one or more converted workflow steps; and associating the one or more converted workflow steps with a portion of a 3D model of equipment within a mixed reality environment; enabling, through the mixed reality environment, access and editing of the 3D model of equipment; and leveraging artificial intelligence-based technology by generating 3D scene information from prompts or extracting visual information from multimedia sources; wherein the legacy document is selected from the group consisting of video, text, voice recordings, and augmented reality; wherein the steps are identified by of one or more methods of identifications consisting of: a user manually selecting the steps, programmatically, observed interactions, and machine learning algorithms; wherein the one or more components are parsed by one or more methods of parsing consisting of artificial intelligence classification or advanced human tool sets; wherein the one or more components is selected from the group of characteristics consisting of: verbs, objects, tools used, and reference images; wherein the legacy document is a plurality of unstructured data; wherein the legacy document is a plurality of interactive modality-derived information; and wherein each of the categorized one or more components comprises a maintenance action, a maintenance tool, and maintenance instructions.

Assignees

• Us Navy
• Us Gov Sec Navy

Inventors

• Greunke Larry Clay

Classifications

• G06F2111/10

  Numerical modelling · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06F2111/18

  using virtual or augmented reality · CPC title

• G06F30/12

  characterised by design entry means specially adapted for CAD, e.g. graphical user interfaces [GUI] specially adapted for CAD · CPC title

• G06T2219/2021

  Shape modification · CPC title