Augmented reality labels systems and methods

Patent metadata

Field	Value
Publication number	US-11328155-B2
Application number	US-201916396351-A
Country	US
Kind code	B2
Filing date	Apr 26, 2019
Priority date	Nov 13, 2015
Publication date	May 10, 2022
Grant date	May 10, 2022

Abstract

Techniques are disclosed for systems and methods for labeling objects displayed by an augmented reality display system used to assist in the operation of mobile structures. Such an augmented reality display system includes a logic device configured to communicate with navigational sensors and imaging modules coupled to a mobile structure, where the navigational sensors are configured to provide navigational data associated with the mobile structure and the imaging module is configured to image a scene from a position on the mobile structure. The logic device is configured to detect an object in the scene, determine a heading reliability associated with the detected object based, at least in part, on the navigational data, and render an integrated model of the scene on a display, where the integrated model is configured to indicate the determined heading reliability associated with the detected object.

First claim

What is claimed is:

1. An apparatus comprising:
a logic device configured to communicate with one or more navigational sensors and an imaging module coupled to a mobile structure, wherein the one or more navigational sensors are configured to provide navigational data associated with the mobile structure and the imaging module is configured to image a scene from a position on the mobile structure, and wherein the logic device is configured to:
detect an object in the scene;
determine a heading and a heading reliability that are associated with the detected object, the heading reliability being a reliability of the determined heading, the heading and heading reliability being determined based, at least in part, on the navigational data; and
render an integrated model of the scene on a display, wherein the integrated model is configured to indicate the determined heading reliability associated with the detected object;
wherein the integrated model comprises an information graphic associated with the detected object; and
the logic device is configured to indicate the determined heading reliability by at least one of:
presence or absence of a reliability indicator with the information graphic, the reliability indicator being present or absent depending on the determined heading reliability;
spatial closeness between the information graphic and the detected object as shown in the rendered integrated model, the spatial closeness depending on the determined heading reliability;
a size of the reliability indicator, the size depending on the determined heading reliability;
a color of the information graphic, the color depending on the determined heading reliability;
a transparency of the information graphic, the transparency depending on the determined heading reliability;
a shape of the information graphic, the shape depending on the determined heading reliability;
an overlay depth of the information graphic relative to at least one other feature displayed by the display, the overlay depth depending on the determined heading reliability.

2. The apparatus of claim 1 , wherein:
the integrated model comprises an image of the scene received from the imaging module, and wherein the logic device is configured to indicate the determined heading reliability associated with the detected object by at least one of:
the presence or absence of the reliability indicator with the information graphic, the reliability indicator being present or absent depending on the determined heading reliability;
the spatial closeness between the information graphic and the detected object as shown in the rendered integrated model, the spatial closeness depending on the determined heading reliability;
the size of the reliability indicator, the size depending on the determined heading reliability;
the transparency of the information graphic, the transparency depending on the determined heading reliability;
the shape of the information graphic, the shape depending on the determined heading reliability;
the overlay depth of the information graphic relative to at least one other feature displayed by the display, the overlay depth depending on the determined heading reliability.

3. The apparatus of claim 1 , wherein:
the logic device is configured to determine the heading reliability is above or equal to a threshold reliability;
the information graphic comprises the reliability indicator configured to indicate the determined heading reliability associated with the detected object; and
the reliability indicator comprises a tail, pointer, and/or spatial link to the detected object to indicate the heading reliability associated with the detected object is above or equal to the threshold reliability.

4. The apparatus of claim 1 , wherein:
the logic device is configured to determine the heading reliability is below a threshold reliability; and
the information graphic is rendered without a tail, pointer, and/or spatial link to the detected object to indicate the heading reliability associated with the detected object is below the threshold reliability.

5. The apparatus of claim 1 , wherein:
the information graphic comprises object information associated with the detected object;
the navigational data comprises navigational data associated with the detected object and/or the mobile structure; and
the object information comprises the navigational data associated with the detected object and/or the mobile structure;
wherein the logic device is configured to indicate the determined heading reliability by at least one of:
the presence or absence of a reliability indicator with the information graphic, the reliability indicator being present or absent depending on the determined heading reliability;
the spatial closeness between the information graphic and the detected object as shown in the rendered integrated model, the spatial closeness depending on the determined heading reliability;
the transparency of the information graphic, the transparency depending on the determined heading reliability;
the shape of the information graphic, the shape depending on the determined heading reliability;
the overlay depth of the information graphic relative to at least one other feature displayed by the display, the overlay depth depending on the determined heading reliability.

6. The apparatus of claim 1 , further comprising the imaging module, wherein the mobile structure comprises a watercraft, and wherein the logic device is configured to:
receive at least one image of the scene from the imaging module; and
render the integrated model of the scene on the display, wherein the integrated model comprises the at least one image of the scene;
wherein the logic device is configured to indicate the determined heading reliability by at least one of:
the presence or absence of a reliability indicator with the information graphic, the reliability indicator being present or absent depending on the determined heading reliability;
the spatial closeness between the information graphic and the detected object as shown in the rendered integrated model, the spatial closeness depending on the determined heading reliability;
the transparency of the information graphic, the transparency depending on the determined heading reliability;
the overlay depth of the information graphic relative to at least one other feature displayed by the display, the overlay depth depending on the determined heading reliability.

7. The apparatus of claim 1 , further comprising:
at least one orientation and/or position sensor (OPS), wherein the logic device is configured to:
receive at least one image of the scene from the imaging module;
receive orientation, position, and/or navigational data corresponding to the at least one image from the OPS and/or the one or more navigational sensors; and
generate at least one horizon-stabilized image corresponding to the received at least one image based, at least in part, on the received orientation, position, and/or navigational data and/or the received at least one image, wherein the integrated model comprises the at least one horizon-stabilized image.

8. The apparatus of claim 7 , wherein the generating the at least one horizon-stabilized image comprises:
determining a slant and elevation of the imaging module corresponding to the at least one received image based, at least in part, on the received orientation, position, and/or navigational data and/or the at least one received image;
rotating pixels of the at least one received image to at least partially compensate for the determined slant;
translating the pixels of the at least one received image to at least partially compensate for the determined elevation; and
cropping the rotated and tran

Assignees

• FLIR Belgium BVBA

Inventors

• Rivers Mark

Classifications

• G01C21/1656

  with passive imaging devices, e.g. cameras · CPC title

• G01C21/203 Primary

  specially adapted for water-borne vessels · CPC title

• G06V20/20 Primary

  in augmented reality scenes · CPC title

• G06T3/60

  Rotation of whole images or parts thereof · CPC title

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title