System for and method of providing a virtual cockpit, control panel, or dashboard using augmented reality

Patent metadata

Field	Value
Publication number	US-9892489-B1
Application number	US-201313971695-A
Country	US
Kind code	B1
Filing date	Aug 20, 2013
Priority date	Aug 20, 2013
Publication date	Feb 13, 2018
Grant date	Feb 13, 2018

Abstract

An apparatus provides a virtual display in an environment for various applications including avionic, naval, military, remote control, medical and other applications. The apparatus includes a camera and a processor system. The processor system for provides the virtual display using an orientation of the head sensed from a camera image. The camera image includes a marker image associated with a fixed marker disposed in the environment. The virtual display includes a virtual panel and an image panel from the camera.

First claim

What is claimed is:

1. An apparatus for providing a virtual display in an environment, the apparatus comprising:
a camera; and
a processor configured to provide a virtual display image using an orientation of a head sensed from a camera image, the camera image including a first marker image and a second marker image, the first marker image associated with a fixed first marker disposed at a first known position in the environment and the second marker image associated with a fixed second marker disposed at a second known position in the environment, wherein at least one flight instrument is disposed in the environment between the first known position and the second known position, wherein the first marker image includes a unique pattern indicating a relative position of the first marker image in the environment with respect to the camera, the processor providing the virtual display image based on the orientation, the orientation based on the relative position of the first marker image and the first and second known positions of the fixed first and second markers, wherein the virtual display image comprises a virtual panel, an image panel from the camera, and an image from a synthetic vision system or from an enhanced vision system.

2. The apparatus of claim 1 , wherein the virtual display image is provided at a virtual location associated with a physical location of the fixed first marker and the fixed second marker, and wherein the virtual display image extends from at least the first marker to the second marker.

3. The apparatus of claim 1 , further comprising: a combiner, the combiner receiving the virtual display image and providing the virtual display image to eyes of a user.

4. The apparatus of claim 3 , wherein the user can see a real world scene above the virtual panel.

5. The apparatus of claim 4 , wherein the camera captures a pattern along a taxiway and the processor provides a visual indication to a pilot corresponding to the pattern along the taxiway, the visual indication comprising information related to an orientation of a vehicle along the taxiway and information related to a location of the vehicle along the taxiway.

6. The apparatus of claim 1 , wherein the camera is one of two or more cameras.

7. The apparatus of claim 1 , wherein the fixed first marker is disposed in a first environment comprising a cockpit, and wherein the fixed second marker is disposed in a second environment external to the cockpit.

8. The apparatus of claim 1 , wherein the fixed first marker and the fixed second marker each include a unique QR code.

9. The apparatus of claim 7 , wherein the processor is configured to display the virtual panel to have a height equal to a physical height of the QR codes associated with the fixed first marker and the fixed second marker.

10. The apparatus of claim 1 , wherein the environment comprises a cockpit, wherein the virtual panel covers a location of at least one non-virtual gauge in the cockpit, wherein the virtual display image covers a structural feature of the cockpit, and wherein the virtual display image comprises a panoramic image comprising the virtual panel and a view of an external environment external to the cockpit.

11. The apparatus of claim 10 , wherein the virtual panel includes a primary flight display.

12. The apparatus of claim 1 , wherein the apparatus is provided in a head mounted display, or eyewear.

13. The apparatus of claim 1 , wherein at least a portion of at least one of the virtual panel and the image panel from the camera displays images provided by a physical primary flight display in a cockpit.

14. A method of providing a virtual display in an environment, the method comprising:
receiving a camera image of the environment using a camera associated with a head, the camera image including a first marker disposed at a first known position in the environment and a second marker disposed at a second known position in the environment, wherein at least one flight instrument is disposed in the environment between the first known position and the second known position;
reading data from a pattern within the first marker indicating a relative position of the first marker with respect to the head;
determining an orientation of the head based on the relative position of the first marker with respect to the head; and
providing a virtual image to the head in response to the orientation of the head and based on the known positions of the first and second markers, the virtual image including a display panel associated with parameters of the environment sensed by equipment in the environment, wherein the virtual image does not block a view through a windshield in the environment, wherein the virtual image is a primary flight display image.

15. The method of claim 14 , wherein the virtual image includes a portion of the camera image.

16. The method of claim 14 , wherein the first marker and the second marker each include a unique QR code, each QR code indicating a respective location within the environment.

17. The method of claim 14 , wherein the virtual image includes a synthetic vision image.

18. The method of claim 14 , wherein the primary flight display image is a representation of a physical primary flight display in a cockpit.

19. An apparatus for providing an optical display, the apparatus comprising:
means for providing image data associated with a virtual head down display, the image data comprising at least one of a synthetic vision system image and an enhanced vision system image representing a physical primary flight display in a cockpit;
means for determining a head orientation using at least one head mounted camera, a fixed first marker disposed at a first known position in the cockpit, and a fixed second marker disposed at a second known position in the cockpit, the fixed first marker including a pattern indicating relative location information of the fixed first marker with respect to the means for providing image data and identification information of the cockpit; and
means for providing the virtual head down display in a virtual location in response to the head orientation and based on the known positions of the first and second markers.

20. The apparatus of claim 19 , wherein the means for providing the virtual head down display comprises a helmet-mounted head up display.

Assignees

• Roggendorf Brian R
• Tyson Stephen E
• Bouchett Steven C
• Rockwell Collins Inc

Inventors

• Roggendorf Brian R
• Tyson Stephen E
• Bouchett Steven C

Classifications

• G06T19/00 Primary

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• G06T3/60 Primary

  Rotation of whole images or parts thereof · CPC title