Generating extended reality overlays in an industrial environment

Patent metadata

Field	Value
Publication number	US-12136174-B1
Application number	US-202117497778-A
Country	US
Kind code	B1
Filing date	Oct 8, 2021
Priority date	Apr 27, 2018
Publication date	Nov 5, 2024
Grant date	Nov 5, 2024

Abstract

A mobile device that includes a camera and an extended reality software application program is employed by a user in an operating environment, such as an industrial environment. The user aims the camera within the mobile device at optical data markers, such as QR codes, that are associated with machines in the environment. The mobile device acquires an image from the camera and decodes the optical data markers included in the acquired image. The mobile device queries the data intake and query system for the values of metrics for the machines associated with the decoded optical data markers. Upon receiving the metric values from the data intake and query system, the mobile device generates AR overlays and superimposes the AR overlays onto the acquired image. The mobile device displays the image with superimposed AR overlays on a display device.

First claim

What is claimed is:

1. A method, comprising:
detecting, within a first image associated with an extended reality (XR) environment, at least a portion of a data marker associated with an object,
wherein at least a portion of the object is present in the first image,
wherein the data marker is physically attached to the object;

acquiring first data associated with the object using the data marker;
determining a size and a plane of the data marker within the first image based on an arrangement of the portion of the data marker in the first image;
comparing the size and the plane of the data marker acquired from the first image with an original size of the data marker to generate a scaling factor;
generating a second image that includes the object and an overlay of a visual representation of the first data acquired using the data marker, wherein an orientation of the overlay within the second image is determined based at least in part on the plane and the overlay is scaled based at least in part on the scaling factor; and
causing the second image to be displayed on a display device.

2. The method of claim 1 , wherein the XR environment is one or more of an augmented reality (AR) environment and a virtual reality (VR) environment.

3. The method of claim 1 , further comprising determining an object identifier based on the data marker and retrieving data or receiving data associated with the object based on the object identifier.

4. The method of claim 1 , wherein the overlay is disposed at least partially within the image relative to the object and a position of the overlay is based at least in part on at least one dimension of the data marker.

5. The method of claim 1 , wherein determining the original size of the data marker comprises retrieving a dimension of the object, wherein the dimension of the object is encoded in the data marker, and wherein the overlay is further based on the dimension of the object.

6. The method of claim 1 , further comprising:
transmitting a query to a server based on an object identifier encoded in the data marker; and
receiving a graphical model of the object in response to the query, wherein the overlay is disposed relative to the object based on at least a portion of the graphical model of the object.

7. The method of claim 1 , further comprising:
transmitting a query to a server based on an object identifier encoded in the data marker; and
receiving event data or real-time data associated with the object in response to the query, wherein the overlay is based on the event data or real-time data.

8. The method of claim 1 , wherein the plane is a two-dimensional plane and wherein the two-dimensional plane is associated with a surface of the data marker and wherein the overlay is disposed relative to the object based on the two-dimensional plane.

9. The method of claim 1 , wherein generating the second image comprises positioning the overlay of the visual representation of the data on the first image relative to the object.

10. The method of claim 1 , wherein generating the second image comprises:
determining a three-dimensional location at which to place the overlay of the visual representation; and
positioning the overlay of the visual representation onto the first image at the three-dimensional location.

11. The method of claim 1 , wherein the original size is encoded in the data marker, and wherein the overlay is disposed relative to the object based on the original size.

12. The method of claim 1 , wherein the size of the data marker is determined using at least one plane detection technique, wherein the plane detection technique uses a position or arrangement of the portion of the data marker, and wherein the overlay is disposed relative to the object based on the size.

13. The method of claim 1 , wherein determining the size of the data marker comprises determining at least one dimension of the data marker based on at least one of a three-dimensional position of the data marker and an orientation of the data marker.

14. The method of claim 1 , wherein an object identifier is encoded in the data marker.

15. The method of claim 1 , further comprising:
transmitting, via a communications network and to a server, a message that includes an object identifier encoded in the data marker; and
receiving, from the server, the data associated with the object, wherein the data is generated by executing a query of machine data, at least a portion of which is available to the server.

16. The method of claim 1 , further comprising:
analyzing a third image that includes at least a portion of the object, to detect one or more edges associated with the object;
generating a plurality of object boundaries based on the one or more edges, wherein each object boundary of the plurality of object boundaries represents a candidate object boundary;
receiving a selection of a first object boundary included in the plurality of object boundaries that represents an actual object boundary; and
storing the first object boundary in a database in association with an object identifier encoded in the data marker.

17. The method of claim 1 , further comprising:
analyzing a third image that includes at least a portion of the object, to detect one or more edges associated with the object;
generating a three-dimensional edge detection model based on the one or more edges;
flattening the three-dimensional edge detection model to generate a flattened two-dimensional boundary;
generating a plurality of object boundaries based on the flattened two-dimensional boundary, wherein each object boundary of the plurality of object boundaries represents a candidate object boundary;
receiving a selection of a first object boundary included in the plurality of object boundaries that represents an actual object boundary; and
storing the first object boundary in a database in association with an object identifier.

18. The method of claim 1 , further comprising:
analyzing a third image that includes at least a portion of the object, to detect one or more edges associated with the object;
generating a plurality of object boundaries based on the one or more edges, wherein each object boundary of the plurality of object boundaries represents a candidate object boundary;
receiving a selection of a first object boundary included in the plurality of object boundaries that represents an actual object boundary;
paint-filling at least a portion of the first object boundary to generate a first paint-filled object boundary;
determining first plane information associated with the first paint-filled object boundary;
generating a three-dimensional model based on at least the first paint-filled object boundary and the first plane information; and
storing the three-dimensional model in a database in association with an object identifier encoded in the data marker.

19. The method of claim 1 , further comprising:
performing a scan of the object to generate a first object boundary;
determining three-dimensional locations associated with the first object boundary;
determining plane information associated with the first object boundary;
generating a three-dimensional model of the object based on the three-dimensional locations and the plane information;
storing the three-dimensional model in a database; and
associating the three-dimensional model with an object identifier encoded in the data marker.

20. The method of claim 1 , further comprising:
performing a scan of the object to generate a first obje

Assignees

• Splunk Inc
• Cisco Tech Inc

Inventors

• Chor Jesse
• Emery Michael
• Chan Christopher
• Wong Glen
• Bhushan Devin

Classifications

• G06V20/20

  in augmented reality scenes · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06K19/06037

  multi-dimensional coding · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06V20/64

  Three-dimensional [3D] objects · CPC title