Secured mobile maintenance and operator system including wearable augmented reality interface, voice command interface, and visual recognition systems and related methods

Patent metadata

Field	Value
Publication number	US-2016124501-A1
Application number	US-201514869166-A
Country	US
Kind code	A1
Filing date	Sep 29, 2015
Priority date	Oct 31, 2014
Publication date	May 5, 2016
Grant date	—

Abstract

Secured remote maintenance, configuration management, and systems engineering apparatuses and methods including wearable augmented reality (AR) interface systems are provided. Embodiments can support secure and remote configuration setting changes (CSC) to a system or subsystem of interest (SSoI) using a head mounted device (HMD), a camera, a visual interface section, and a processing section including a plurality of processing instructions operable to operate a command input (CI) interface using one or more user command input CIs (e.g. voice or motion/gesture input recognition) using a secure user authentication systems. HMD machine vision and pattern recognition systems visually identify a SSoI, displaying a 3D model(s) of the SSoI on the HMD's visual interface using the AR interface system, obtaining a plurality of SSoI data, and displaying one or more of the SSoI data in relation to the 3D model on the HMD visual interface to support various tasks including CI directed CSCs.

First claim

1 . A secured mobile maintenance and operator system comprising:
a head mounting device (HMD) comprising a plurality of components wherein said components comprise:
a microphone;
an image capture system oriented towards a user field of view (FOV);
a HMD visual interface comprising a transparent section positioned over said user FOV and a visual interface system configured to display a plurality of visual elements including a graphical user interface onto said transparent section;
a processing section including a plurality of machine readable processing instructions operable to operate said plurality of components and a command input interface using one or more user command inputs (CI) comprising voice inputs or motion/gesture input recognition;
a first transceiver and input/output interface configured to communicate with a second transceiver system;
a secure user authentication system (SUAS) configured to receive user access requests as one of said voice or gesture inputs;
a machine vision and pattern recognition systems configured to visually identify a system or subsystem of interest (SSoI) in said FOV; and
an augmented reality (AR) control section operable to generate a configuration interface display configured for displaying at least one 3D model of the SSoI on the HMD visual interface, orienting the 3D model to align and scale over the SSoI in said FOV on said transparent section based on orientation, position, distance, and perspective of said transparent section in relation to said SSoI, obtaining a plurality of SSoI data through said transceiver and input/output section from a remote server comprising SSoI configuration data, displaying one or more of the SSoI data in relation to the 3D model on the HMD visual interface, orienting the SSoI data to align with over top of or next to components of the SSoI displayed on the transparent section of the HMD visual interface;
wherein one or more of said processing instructions comprise machine readable instructions to execute a change of a configuration setting of a selected SSoI based on one or more said voice inputs or motion/gesture inputs associated with selection and changing one or more configuration settings of a selected said SSoI displayed in said HMD visual interface;
wherein one or more of said processing instructions comprise machine readable instructions to execute a change of the HMD visual interface to display one or more said SSoI data based on user's selection.

2 . A system as in claim 1 , further comprising said second transceiver, a network system coupled to said second transceiver, and said remote server communicatively coupled with said network system.

3 . A secured mobile maintenance and operator system comprising:
a head mounted device (HMD) including a plurality of components comprising:
a visual interface system comprising a visual overlay display system comprising a transparent element configured to generate visual graphical user interfaces (GUI) positioned within a field of view (FOV) of a wearer of said HMD on said transparent element;
at least one video camera configured to record visual inputs oriented to capture images in at least said FOV;
a microphone configured to receive aural inputs in proximity to the HMD comprising user voice command inputs (CI);
a position and orientation section configured to determine position and orientation of the HMD with respect to a reference framework comprising a layout or map of an internal area of a structure and predetermined structures within in said internal area;
a machine vision and pattern recognition system;
an augmented reality (AR) section configured to generate one or more three dimensional (3D) models of a system or subsystem of interest (SSoI) on said visual interface;
a CI control section configured to control one or more CI interfaces and said microphone;
a wireless communication system configured to communicate with said one or more remote data servers and with said one or more SSoI;
a user authentication system (UAS) interface configured to authenticate a user's access credentials and allow access with said one or more remote systems comprising maintenance, configuration management, or systems engineering data stores using the wireless communication system and generate user authentication interfaces and input interfaces for user input of authentication data and enabling access to said one or more remote data servers; and
a control section comprising a memory and machine processing control element configured to operate said HMD including said plurality of components and control operation of said machine vision and recognition system, perform data storage and data recall functions with said plurality of components including said memory and said one or more remote data servers, said AR section configured to selectively superimpose at least one said GUI comprising a plurality of computer-generated images on said transparent element comprising a visual overlay display on or within said FOV thus providing a composite view, and said CI control section, said wireless communication system, and said UAS interface;
wherein said control section is configured to execute a local user login process to request the user to input a first authentication data to be granted access to the HMD, wherein the control section is configured to operate said one or more video camera with said machine vision and pattern recognition system that is configured to selectively pattern match identify said SSoI based on said CI or an automatic pattern match identification mode for SSoI's matched with remote pattern matching data stored in said one or more remote data servers or within a local pattern matching data stored in said memory, displaying one or more elements of said 3D model of the selected SSoI on the one or more visual interface of the HMD over the SSoI using said AR section, communicating with SSoI or a SSoI control or monitoring system (SSoICoMs) to obtain a plurality of SSoI data with the SUAS, and displaying one or more of the SSoI data in relation to the 3D model on the HMD visual interface; wherein the control section is configured to operate the HMD to obtain and display a comparison of said plurality of SSoI data from one or more said SSoIs or said SSoICoMs for SSoIs in said FOV including SSoI operational status data including current operational state and actual SSoI configurable parameters with one or a plurality of predetermined SSoI control settings used to control said SSoI including default configurable parameter settings associated with one of a plurality of SSoI operational modes including startup, operations including power levels, and maintenance operational states; wherein the control section is configured to operate the HMD and said UAS to obtain and display said comparison of said plurality of SSoI data from one or more said SSoIs or said SSoICoMs for SSoIs in said FOV upon pattern recognition of one or more said SSoIs within said FOV using said machine vision and pattern recognition system or upon receiving a CI request for a change of at least one configurable parameter of the SSoI; wherein said HMD is configured to execute an authentication process for the CI request with user access data using one or more authentication modes comprising said user access credentials comprising a user name and password, a stored PKI certificate data with a user code or pin.

4 . The system as in claim 3 , wherein said one or more CI interfaces comprises one or more interfaces comprising voice, keyboard, motion, or gesture recognition CI interface control sections.

5 . The system as in claim 3 , wherein said machine vision and pattern recognition system that is configured to selectively pattern match identify said SSoI identifies said SSoI by identifying a bar code attached to the SSoI.

Assignees

• Us Navy

Inventors

• Lam Jack
• Navarro Philippe

Classifications

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G06F3/048

  Interaction techniques based on graphical user interfaces [GUI] · CPC title