Seamless bridging ar-device and ar-system

Patent metadata

Field	Value
Publication number	US-2018357828-A1
Application number	US-201816005563-A
Country	US
Kind code	A1
Filing date	Jun 11, 2018
Priority date	Jun 12, 2017
Publication date	Dec 13, 2018
Grant date	—

Abstract

An Augmented Reality (AR)-device, including a visual sensor for capturing an environment of the AR-device, a display providing a real view of the environment, and overlays onto the real view according to AR-data including at least one of design data and measured data, a computer for reading and controlling the visual sensor, at least one of receiving, generating and storing the AR-data, and identification features which are assigned to each of a plurality of reference systems, generating the overlays, establishing a referenced status of the AR-device relative to each of the reference systems by identifying a reference system based on at least one identification feature captured by the visual sensor, determining a pose of the AR-device relative to the identified reference system, and maintaining the referenced status in case of loss of visual contact between the visual sensor and the identification feature, based on a Visual Localisation and Mapping (VSLAM)-process.

First claim

1 . An Augmented Reality (AR)-device comprising:
a visual sensor configured for capturing an environment of the AR-device, a display configured for providing:
a real view of the environment, and
overlays onto the real view according to AR-data, wherein the AR-data comprise at least one of design data and measured data,
a computer configured for:
reading and controlling the visual sensor,
at least one of receiving, generating and storing
the AR-data, and
identification features which are assigned to each of a plurality of reference systems,

generating the overlays,
establishing a referenced status of the AR-device relative to each of the reference systems in each case by:
identifying a reference system based on at least one identification feature captured by the visual sensor, and
determining a pose of the AR-device relative to the identified reference system, and

maintaining the referenced status even in case of loss of visual contact between the visual sensor and the at least one identification feature, based on a Visual Localisation and Mapping (VSLAM)-process.

2 . The AR-device according to claim 1 ,
wherein the VSLAM-process and establishing the referenced status are based at least on the output of the visual sensor.

3 . The AR-device according to claim 1 , further comprising:
a position and orientation sensor configured to provide position and orientation data for the computer, wherein the VSLAM-process is further based on the position and orientation data.

4 . An Augmented Reality (AR)-system comprising
an AR-device comprising:
a visual sensor configured for capturing an environment of the AR-device,
a display configured for providing:
a real view of the environment, and
overlays onto the real view according to AR-data, wherein the AR-data comprise at least one of design data and measured data,

a computer configured for:
reading and controlling the visual sensor,
at least one of receiving, generating and storing
(i) the AR-data, and
(ii) identification features which are assigned to each of a plurality of reference systems,
generating the overlays,
establishing a referenced status of the AR-device relative to each of the reference systems in each case by:
(i) identifying a reference system based on at least one identification feature captured by the visual sensor, and
(ii) determining a pose of the AR-device relative to the identified reference system, and

maintaining the referenced status even in case of loss of visual contact between the visual sensor and the at least one identification feature, based on a Visual Localisation and Mapping (VSLAM)-process; and
at least one surveying instrument, each defining one of the plurality of reference systems.

5 . The AR-system according to claim 4 ,
wherein at the least one surveying instrument is configured to generate and provide to the AR-device at least one of
the AR-data, and
the at least one identification feature.

6 . The AR-system according to claim 4 ,
wherein the identification features are one or more of
image data,
a coded marker pattern, or
landmark coordinates.

7 . A method for establishing and seamlessly bridging a referenced status of an Augmented Reality (AR)-device, the method comprising:
capturing an environment of the AR-device using a visual sensor of the AR-device; at least one of receiving, generating and storing identification features which are assigned to each of a plurality of reference systems; generating overlays onto a real view of the environment displayed on a display of the AR-device, wherein the AR-data comprise at least one of design data and measured data; establishing a referenced status of the AR-device relative to each of the reference systems in each case by:
identifying a reference system based on at least one identification feature captured by the visual sensor, and
determining a pose of the AR-device relative to the identified reference system;
identifying a first reference system based on at least one of first identification features captured by the visual sensor; and based on a Visual Localisation and Mapping (VSLAM)-process, maintaining the referenced status even in case none of the at least one first identification feature is visible to the visual sensor.

8 . The method according to claim 7 , further comprising:
identifying a second reference system based on at least one of second identification features captured by the visual sensor, and by determining a pose of the AR-device relative to the second identification feature, establishing a referenced status of the AR-device with respect to the second reference system.

9 . The method according to claim 8 , further comprising:
spatially interrelating the first identification features of the first reference system with the second identification features of the second reference system by generating according transformation parameters.

10 . The method according to claim 9 ,
wherein generating the transformation parameters is based on the VSLAM-process.

11 . The method according to claim 10 , wherein the referenced status with respect to the first reference system is maintained even in case none of the first identification features is visible to the visual sensor, and wherein the referenced status with respect to the second reference system is maintained even in case none of the second identification features is visible to the visual sensor.

12 . The method according to claim 10 ,
wherein at least one of
determining the pose of the AR-device relative to the first identification features, and
determining the pose of the AR-device relative to the second identification features
comprises using an image processing algorithm from photogrammetry or computer vision, in particular using a VSLAM-process.

13 . The method according to claim 10 ,
wherein at least one of
determining the pose of the AR-device relative to the first identification features and
determining the pose of the AR-device relative to the second identification features
comprises using supportive position and orientation data obtained by the position and orientation sensor comprised by the AR-device.

14 . The method according to claim 9 , further comprising
generating an indication based on
at least one of the first indication features, or at least one of the second indication features,
the position of the origin of the first reference system or of the second reference system, and
providing the indication on the display of the AR-device.

15 . The method according to claim 14 ,
wherein the indication comprises at least one of
an arrow pointing in the direction of an according origin of a reference system or in the direction of an identification feature assigned to an according reference system, and
a distance value of the distance between the AR-device and an according origin of a reference system or of the distance between the AR-device and an identification feature assigned to an according reference system.

Assignees

• Hexagon Technology Ct Gmbh

Inventors

• Metzler Bernhard
• Havnor Martin

Classifications

• G06T7/292

  Multi-camera tracking · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T7/73

  using feature-based methods · CPC title