Interactive control station

Patent metadata

Field	Value
Publication number	US-2017316613-A1
Application number	US-201515527817-A
Country	US
Kind code	A1
Filing date	Nov 9, 2015
Priority date	Nov 19, 2014
Publication date	Nov 2, 2017
Grant date	—

Abstract

A mixed reality system for creating a terminal control station enabling communication with and/or control of remote functions and applications, the system comprising a headset ( 100 ) for placing over a user's eyes, in use, said headset including a screen ( 102 ), the system further comprising a processor ( 104 ) configured to receive data from multiple sources and display said data on said screen within a three-dimensional virtual environment, and an input for receiving control information data from an external data source within the real world environment, said processor ( 104 ) being further configured to receive image data representative of said external data source and blend said image data into said three-dimensional virtual environment to create a mixed reality environment, including a representation of said external data source and said control information data, to be displayed on said screen ( 102 ), the system being configured to allow a user, in use, to manipulate data displayed within said mixed reality environment.

First claim

What is claimed is:

1 . A mixed reality system for creating a terminal control station enabling communication with and/or control of remote functions and applications, the system comprising:
a headset for placing over a user's eyes, in use, said headset including a screen, the system further comprising: a processor configured to receive data from multiple sources and display said data on said screen within a three-dimensional virtual environment; a pair of spatially separated image capture devices for capturing respective images of a real world environment in a vicinity of the user; and an input for receiving control information data from an external data source within the real world environment; said processor being further configured to receive image data representative of said external data source and blend said image data into said three-dimensional virtual environment to create a mixed reality environment, including a representation of said external data source and said control information data, to be displayed on said screen, the system being configured to allow a user, in use, to manipulate data displayed within said mixed reality environment.

2 . The system according to claim 1 , wherein the terminal control station comprises physical components associated therewith, and said processor is preconfigured to display at least one physical component of said terminal control station, and data associated therewith, on said screen within said three-dimensional virtual environment.

3 . The system according to claim 1 , configured to allow a user, in use, to manipulate data displayed within said mixed reality environment by means of hand gestures.

4 . The system according to claim 3 , further comprising a control panel or keyboard in respect of which a user can, in use, manually indicate data selection and manipulation commands, wherein said processor is configured to identify from said captured images said data selection and manipulation commands and generate control signals representative thereof.

5 . The system according to claim 1 , wherein said processor is preconfigured to identify, from said captured images, at least one physical component of said terminal control station within said real world environment, and blend image data representative of said physical component into said three-dimensional virtual environment.

6 . The system according to claim 1 , said processor being further configured to define a depth map using respective image frame pairs to produce three dimensional image data.

7 . The system according to claim 6 , wherein said image capture devices are mounted on said headset.

8 . The system according to claim 7 , wherein said image capture devices are mounted on said headset so as to be substantially aligned with a user's eyes, in use.

9 . The system according to claim 1 , configured to allow a user, in use, to select objects, portions or people from their real world environment to be included within said mixed reality environment by means of one or more predefined hand gestures.

10 . A method of providing a terminal control station enabling communication with and/or control of remote functions and applications, the method comprising:
providing a mixed reality system according to claim 1 ; configuring the processor to receive data from multiple sources and display said data on said screen within a three-dimensional virtual environment; receiving control information data from an external data source within the real world environment; receiving image data representative of said external data source; and blending said image data into said three-dimensional virtual environment to create a mixed reality environment, including a representation of said external data source and said control information data, to be displayed on said screen.

11 . The method according to claim 10 , wherein said terminal control station is an aircraft or Unmanned Aerial Vehicle ground station, and the method includes the step of configuring the processor to receive image data representative of the real world environment, identify, from said image data, at least one physical component of said terminal control station within said real world environment, and blend image data representative of said physical component into said three-dimensional virtual environment.

Assignees

• Bae Systems Plc

Inventors

• Wright Julian David
• Colosimo Nicholas Giacomo Robert
• Whiteford Christopher James
• Page Heather Jean
• Goodall Mark Robert

Classifications

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/0425

  using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386) · CPC title