Augmented and virtual reality based process oversight

Patent metadata

Field	Value
Publication number	US-10454943-B2
Application number	US-201615234786-A
Country	US
Kind code	B2
Filing date	Aug 11, 2016
Priority date	Aug 17, 2015
Publication date	Oct 22, 2019
Grant date	Oct 22, 2019

Abstract

The present disclosure involves systems and computer implemented methods for generating a virtual or augmented reality-based oversight system of business processes at a location remote from where the business process is executed. In one example, system operations include identifying a process-based operation performed at a physical first location, where the operation is associated with a request for oversight from a remote second location. A current state of the operation at the first location is identified, and at least one input source at the first location is identified. A simulation of the current state of the operation at the first location incorporating input from the input source(s) is generated for presentation at the second location. An action performed at the first location in the current state of the operation is identified, and the second location is then transmitted a request to authorize the identified action.

First claim

What is claimed is:

1. A system comprising:
a memory;
at least one hardware processor interoperably coupled with the memory and configured to:
identify, at a first system executing a process-based operation performed at a physical first location in association with a customer, a current action of the process-based operation performed by a first user at the first location for the customer, the current action of the process-based operation triggering an automatic request for oversight to be performed by a second system at a remote second location;
in response to the identifying, identify a current state associated with a current step of the process-based operation at the first location;
determine at least one image or video input source capturing at least a portion of the process-based operation in the current state occurring at the first location;
generate, for presentation at a display at the second location, a simulation of the current state of the process-based operation at the first location, the simulation incorporating input from the at least one determined image or video input source and simulating the current state of the process-based operation performing the current action of the process-based operation from a perspective of the first user;
identify an action performed at the first location in the current state of the process-based operation, wherein the action performed at the first location seeks authorization from the second location; and
transmit a request to the second location to authorize the identified action.

2. The system of claim 1 , wherein the physical first location is associated with a first user, and wherein the remote second location is associated with a second user.

3. The system of claim 2 , the at least one hardware processor configured to:
receive authorization from the second user at the second location; and
in response to receiving authorization from the second user at the second location, notify the first user at the first location of the authorization and allowing the process-based operation to continue to a next state at the first location.

4. The system of claim 2 , the at least one hardware processor configured to: in response to not receiving authorization from the second user at the second location, suspending execution of the process-based operation at the first location.

5. The system of claim 4 , wherein suspending execution of the process-based operation at the first location includes providing control of the process-based operation to the second user at the second location, wherein the second user can perform or revise the identified action within the simulation of the current state, wherein feedback representing the performed or revised actions performed by the second user at the second location are transmitted to a display associated with the first user at the first location.

6. The system of claim 4 , wherein suspending execution of the process-based operation at the first location includes:
providing control of the process-based operation to the second user at the second location;
receiving instructions from the second user related to at least one correction to the identified action; and
providing control of the process-based operation to the first user at the first location to revise the identified action based on the received instructions.

7. The system of claim 2 , wherein identifying the process-based operation includes:
identifying, from a plurality of process-based operations performed at a plurality of physical locations by at least the first user and a third user, a queue of two or more requests for oversight of at least two process-based operations performed by at least one of a second user and a fourth user remote from at least the first user and the third user, wherein the identified process-based operation is associated with the remote second user for oversight.

8. The system of claim 2 , wherein identifying the at least one input source associated with the current state of the process-based operation at the first location of the first user includes identifying one of a video or image feed capturing data at the location of the first user, a video or image feed associated with the first user, a video or image feed associated with the process-based actions of the first user at the first location, or a visualization of a display associated with the first user at the first location.

9. The system of claim 2 , wherein identifying the at least one input source associated with the current state of the process-based operation at the first location of the first user includes a state indication associated with at least one connected device present at the first location or a state indication associated with at least one connected device involved in the process-based operation.

10. The system of claim 2 , wherein the first user is not notified of the automatic request for oversight from the remote second user.

11. The system of claim 2 , the at least one hardware processor configured to: display, to the second user at the second location, the generated simulation of the current state of the process-based operation.

12. The system of claim 11 , wherein the generated simulation of the current state of the process-based operation is displayed at the second location to the second user via an augmented reality display, wherein the augmented reality display comprises an overlay of at least a portion of input from the at least one input source on real-world content at the second location, and wherein the augmented reality display allows for input from the second user to authorize actions in the process-based operation.

13. The system of claim 11 , wherein the generated simulation of the current state of the process-based operation is displayed at the second location to the second user via virtual reality display, wherein the virtual reality display is presented within a virtual reality output device at the second location, and wherein the virtual reality display allows for input from the second user to authorize actions in the process-based operation.

14. A non-transitory, computer-readable medium storing computer-readable instructions executable by a computer and configured to:
identify, at a first system executing a process-based operation performed at a physical first location in association with a customer, a current action of the process-based operation performed by a first user at the first location for the customer, the current action of the process-based operation triggering an automatic request for oversight to be performed by a second system at a remote second location;
in response to the identifying, identify a current state associated with a current step of the process-based operation at the first location;
determine at least one image or video input source capturing at least a portion of the process-based operation in the current state occurring at the first location;
generate, for presentation at a display at the remote second system, a simulation of the current state of the process-based operation at the first location, the simulation incorporating input from the at least one identified input source;
generate, for presentation at a display at the second location, a simulation of the current state of the process-based operation at the first location, the simulation incorporating input from the at least one determined image or video input source and simulating the current state of the process-based operation performing the current action of the process-based operation from a perspective of the first user;
identify an action performed at the first location in the current s

Assignees

• Toronto Dominion Bank

Inventors

• Moretti Christianne
• Pryor James Zachary
• Hamilton Matthew
• Barnett Jonathan K
• Chan Paul Mon-Wah
• Del Vecchio Orin
• Barbon John
• Lee John Jong-Suk

Classifications

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06Q10/103

  Workflow collaboration or project management · CPC title

• H04L63/107 Primary

  wherein the security policies are location-dependent, e.g. entities privileges depend on current location or allowing specific operations only from locally connected terminals · CPC title

• G02B27/01

  Head-up displays · CPC title

• G06F3/14 Primary

  Digital output to display device {; Cooperation and interconnection of the display device with other functional units} · CPC title