Augmented reality-based measuring system

Patent metadata

Field	Value
Publication number	US-10878633-B2
Application number	US-201916406912-A
Country	US
Kind code	B2
Filing date	May 8, 2019
Priority date	May 8, 2018
Publication date	Dec 29, 2020
Grant date	Dec 29, 2020

Abstract

An Augmented Reality (AR)-based system includes a mobile sensor, AR device, coordinate measuring machine, and computer. The AR-device includes a camera for capturing an environment, and a display providing a view of the environment and overlays. The coordinate measuring instrument determines a position of the mobile sensor and the AR-device. The computer: controls the camera, provides a reciprocal data connection, provides the AR-data and at least one corresponding identification feature, establishes a referenced status of the AR-device by, in each case, identifying a respective identification feature captured by the camera, determines a pose of the AR-device relative to the respective identification feature, and generates the overlays in the referenced status based on the pose of the AR-device.

First claim

What is claimed is:

1. An Augmented Reality (AR)-based system comprising:
a mobile sensor configured for measuring an object and for being maneuverable;
an AR-device comprising:
a camera configured for capturing an environment of the AR-device, and
a display configured for providing:
a real view of the environment, and
overlays onto the real view according to AR-data, said AR-data being spatially associated with the object,

a coordinate measuring instrument configured for determining a position of the mobile sensor and a position of the AR-device; and
a computer configured for:
reading and controlling the camera,
providing a reciprocal data connection among the AR-device and the coordinate measuring instrument,
providing the AR-data and, for each of the mobile sensor, the coordinate measuring instrument, and the object, at least one corresponding identification feature,
establishing a referenced status of the AR-device relative to each of the mobile sensor, the coordinate measuring instrument, and the object by, in each case,
identifying a respective identification feature captured by the camera,
determining a pose of the AR-device relative to the respective identification feature based at least in part on the position of the AR-device determined by the coordinate measuring instrument,

maintaining the referenced status of the AR-device relative to at least one of the mobile sensor, the coordinate measuring instrument, and the object in case of loss of visual contact between the camera and the respective identification feature, based on a Visual Localisation and Mapping (VSLAM)-process making use of structure of the environment, and
generating the overlays in the referenced status based on the pose of the AR-device relative to at least one of the mobile sensor, the coordinate measuring instrument, and the object.

2. The AR-based system according to claim 1 , further comprising:
a vehicle, wherein the mobile sensor is maneuverable by being arranged on said vehicle.

3. The AR-based system according to claim 1 , wherein the AR-data are based on measuring instructions provided by the computer.

4. The AR-based system according to claim 3 , wherein the measuring instructions comprise at least one of:
indications where to place or align the mobile sensor, and
instructions on how to perform the measurement.

5. The AR-based system according to claim 1 , wherein the AR-data is based on measurement data of the object, said measurement data obtained with at least one of the mobile sensor and the coordinate measuring instrument.

6. The AR-based system according to claim 5 , wherein the measurement data comprise a report based on at least one of
quantitative measurement results, and
qualitative measurement results.

7. The AR-based system according to claim 1 , wherein the AR-data are based on fitting instructions, which are based on measurement data of the object, said measurement data obtained by at least one of the mobile sensor and the coordinate measuring instrument.

8. The AR-based system according to claim 1 , wherein the mobile sensor is:
embodied as one of a measuring probe, articulated arm coordinate measuring machine, a scanner, and a multi-camera device, and
configured for providing a light pattern, and

wherein the coordinate measuring instrument is configured for:
detecting the light pattern, and
determining the pose of the mobile sensor based on the detected light pattern.

9. The AR-based system according to claim 1 , wherein the coordinate measuring instrument is embodied as one of a laser tracker, a laser scanner, a camera system, an articulated arm coordinate measuring machine, a tachymeter, a theodolite, a total station, a white light sensor, a search of light sensor, a time of flight measuring system, or a laser radar system.

10. The AR-based system according to claim 1 , wherein the AR-device is embodied as one of a tablet computer, a smart phone, AR-glasses, an AR-helmet, one or more contact lens projectors, or a virtual retinal display.

11. The AR-based system according to claim 1 , wherein the computer is comprised by at least one of the mobile sensor, the coordinate measuring instrument, the AR-device, and an external computing device.

12. The AR-based system according to claim 1 , wherein at least one of the mobile sensor, the coordinate measuring instrument, and the AR-device comprise a wireless communication module allowing for exchanging data.

13. The AR-based system according to claim 1 , wherein each identification feature is embodied as at least one of a coded pattern, a 3D-CAD-model, and an image.

14. A method of providing visual support for a user using a system, the method comprising:
providing an Augmented Reality (AR)-based system comprising:
a mobile sensor configured for measuring an object and for being maneuverable manoeuvrable;
an AR-device comprising:
a camera configured for capturing an environment of the AR-device, and
a display configured for providing:
a real view of the environment, and
overlays onto the real view according to AR-data, said AR-data being spatially associated with the object, and

a coordinate measuring instrument configured for determining a position of the mobile sensor and a position of the AR-device; and
a computer configured for:
reading and controlling the camera,
providing a reciprocal data connection among the AR-device and the coordinate measuring instrument,
providing the AR-data and, for each of the mobile sensor, the coordinate measuring instrument, and the object, at least one corresponding identification feature,
establishing a referenced status of the AR-device relative to each of the mobile sensor, the coordinate measuring instrument, and the object by, in each case,
identifying a respective identification feature captured by the camera,
determining a pose of the AR-device relative to the respective identification feature based at least in part on the position of the AR-device determined by the coordinate measuring instrument, and
maintaining the referenced status of the AR-device relative to at least one of the mobile sensor, the coordinate measuring instrument, and the object in case of loss of visual contact between the camera and the respective identification feature, based on a Visual Localisation and Mapping (VSLAM)-process making use of structure of the environment, and
generating the overlays in the referenced status based on the pose of the AR-device relative to at least one of the mobile sensor, the coordinate measuring instrument, and the object; and

providing a reciprocal data connection among the AR-device and the coordinate measuring instrument;
providing the AR-data and, for each of the mobile sensor, the coordinate measuring instrument, and the object, at least one corresponding identification feature;
establishing a referenced status of the AR-device relative to each of the mobile sensor, the coordinate measuring instrument, and the object by, in each case:
identifying a respective identification feature captured by the camera,
determining a pose of the AR-device relative to the respective identification feature based at least in part on the position of the AR-device determined by the coordinate measuring instrument, and

generating the overlays in the referenced status based on the pose of the AR-device relative to at least one of the mobile sensor, the coordinate measuring instrument, and the object.

15. An Augmented Reality (AR)-based system comprising:
a mobile sensor configured for measuring an object and for being maneuverable;
an AR-device comp

Assignees

• Leica Geosystems Ag

Inventors

• Steiner Markus
• Lettau Michael

Classifications

• G06V20/20

  in augmented reality scenes · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G01B5/008

  using coordinate measuring machines · CPC title

• G01B11/005

  coordinate measuring machines · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title