Method and system for implementing augmented reality (AR)-based assistance within work environment

Patent metadata

Field	Value
Publication number	US-11847751-B2
Application number	US-202217867565-A
Country	US
Kind code	B2
Filing date	Jul 18, 2022
Priority date	May 22, 2018
Publication date	Dec 19, 2023
Grant date	Dec 19, 2023

Abstract

Novel tools and techniques are provided for implementing augmented reality (AR)-based assistance within a work environment. In various embodiments, a computing system might receive, from a camera having a field of view of a work environment, first images of at least part of the work environment, the first images overlapping with a field of view of a user wearing an AR headset; might analyze the received first images to identify objects; might query a database(s) to determine a task associated with a first object(s) among the identified objects; might generate an image overlay providing at least one of graphical icon-based, text-based, image-based, and/or highlighting-based instruction(s) each indicative of instructions presented to the user to implement the task associated with the first object(s); and might display, to the user's eyes through the AR headset, the generated first image overlay that overlaps with the field of view of the user's eyes.

First claim

What is claimed is:

1. An apparatus comprising:
at least one hardware processor; and
a non-transitory computer readable medium having stored thereon machine readable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to:
receive, from an image capture device, at least one image of a portion of an environment, the image overlapping with a field of view of a user wearing an augmented reality (“AR”) headset that includes the image capture device;
analyze the at least one image to identify a plurality of objects disposed in the environment;
determine a task associated with at least one object of the identified plurality of objects,
wherein the task includes a plurality of steps associated with a particular type of task, and
wherein the plurality of steps includes specific task-related step-by-step actions based on meeting of a specified percentage condition of usage that includes partial usage associated with the identified plurality of objects;

generate an image overlay that provides at least one instruction to be presented to the user to implement the plurality of steps; and
generate, to the user through the AR headset, a display of the generated image overlay.

2. The apparatus according to claim 1 , wherein the plurality of steps includes:
specific timings for the specific task-related step-by-step actions associated with the identified plurality of objects; and
specific sequences for the specific task-related step-by-step actions associated with the identified plurality of objects.

3. The apparatus according to claim 1 , wherein each object of the plurality of objects includes a radio frequency identifier (“RFID”) tag affixed thereto.

4. The apparatus according to claim 3 , wherein the machine readable instructions to analyze the at least one image to identify the plurality of objects disposed in the environment, when executed by the at least one hardware processor, further cause the at least one hardware processor to:
identify each object of the plurality of objects based on the affixed RFID tag in conjunction with identifying each object of the plurality of objects based on analysis of the at least one image; and
filter, based on a gesture or a selection by the user, RFID tag information associated with an object outside of a central portion of the field of view of the user or outside of an area defined by the gesture or the selection by the user.

5. The apparatus according to claim 1 , wherein the AR headset includes at least one of a set of AR goggles, a pair of AR-enabled eyewear, an AR-enabled smartphone mounted in a headset, or an AR helmet.

6. The apparatus according to claim 1 , wherein the AR headset includes at least one of at least one earpiece or at least one speaker, and wherein the machine readable instructions, when executed by the at least one hardware processor, further cause the at least one hardware processor to:
generate at least one audio-based instruction corresponding to the at least one instruction to be presented to the user.

7. The apparatus according to claim 1 , wherein the at least one image represents at least one first image, and wherein the machine readable instructions, when executed by the at least one hardware processor, further cause the at least one hardware processor to:
receive, from a further image capture device that is disposed external to the AR headset, at least one second image of the portion of the environment, the at least one second image overlapping with the field of view of the user wearing the AR headset,
wherein the machine readable instructions to analyze the at least one first image to identify the plurality of objects disposed in the environment, when executed by the at least one hardware processor, further cause the at least one hardware processor to:
analyze the at least one first image and the at least one second image to identify the plurality of objects disposed in the environment.

8. The apparatus according to claim 1 , wherein the machine readable instructions, when executed by the at least one hardware processor, further cause the at least one hardware processor to:
at least one of superimpose the at least one instruction over the at least one object, display the at least one instruction around the at least one object, or display the at least one instruction beside the at least one object.

9. The apparatus according to claim 1 , wherein the at least one instruction includes at least one of a graphical icon-based instruction, a text-based instruction, an image-based instruction, or a highlighting-based instruction.

10. The apparatus according to claim 1 , wherein the machine readable instructions, when executed by the at least one hardware processor, further cause the at least one hardware processor to:
illuminate, by a light source, at least one of the at least one object or the portion of the environment.

11. The apparatus according to claim 1 , wherein the machine readable instructions, when executed by the at least one hardware processor, further cause the at least one hardware processor to:
determine whether an object of the plurality of objects is not identifiable; and
based on a determination that the object of the plurality of objects is not identifiable, generate the image overlay to include the at least one instruction to prompt the user to specifically view the object that is not identifiable.

12. The apparatus according to claim 1 , wherein the machine readable instructions, when executed by the at least one hardware processor, further cause the at least one hardware processor to:
generate the image overlay to highlight an area in the environment in which the at least one object is to be fitted.

13. A method comprising:
receiving, by at least one hardware processor, at least one image of a portion of an environment, the image overlapping with a field of view of a user wearing an augmented reality (“AR”) headset;
analyzing, by the at least one hardware processor, the at least one image to identify a plurality of objects disposed in the environment;
determining, by the at least one hardware processor, a task associated with at least one object of the identified plurality of objects,
wherein the task includes a plurality of steps associated with a particular type of task, and
wherein the plurality of steps includes:
specific task-related step-by-step actions based on meeting of a specified percentage condition of usage that includes partial usage associated with the identified plurality of objects;
specific timings for the specific task-related step-by-step actions associated with the identified plurality of objects; and
specific sequences for the specific task-related step-by-step actions associated with the identified plurality of objects;

generating, by the at least one hardware processor, an image overlay that provides at least one instruction to be presented to the user to implement the plurality of steps; and
generating, by the at least one hardware processor, to the user through the AR headset, a display of the generated image overlay.

14. The method according to claim 13 , wherein the at least one image represents at least one first image, and wherein analyzing, by the at least one hardware processor, the at least one image to identify the plurality of objects disposed in the environment further comprises:
receiving, by the at least one hardware processor, from an image capture device that is disposed external to the AR headset, at least one second image of the portion of the environment, the at least one second image overlapping with the field of

Assignees

• Agilent Technologies Inc

Inventors

• Ben-Dor Amir
• Arbel Elad
• Workman Richard
• Lim Victor

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G02B27/017

  Head mounted · CPC title

• G06K7/10396

  the interrogation device being wearable, e.g. as a glove, bracelet, or ring (wearable aerials in general H01Q1/27) · CPC title

• G06K7/10881

  constructional details of hand-held scanners · CPC title

• G06K7/1408

  the method being specifically adapted for the type of code · CPC title