Tracking objects during processes

Patent metadata

Field	Value
Publication number	US-9911351-B2
Application number	US-201414192337-A
Country	US
Kind code	B2
Filing date	Feb 27, 2014
Priority date	Feb 27, 2014
Publication date	Mar 6, 2018
Grant date	Mar 6, 2018

Abstract

Embodiments are disclosed that relate to tracking one or more objects during a process that utilizes the objects. For example, one embodiment provides a method for monitoring performance of a process involving one or more objects, wherein the method includes receiving a set of rules defining one or more portions of the process and receiving object identification information regarding the one or more objects. The method further includes, for a selected portion of the process, receiving image information of a physical scene, identifying from the image information and the object identification information an operation performed with an identified object in the physical scene, and taking an action based upon whether the operation satisfies a rule of the set of rules associated with the selected portion of the process.

First claim

The invention claimed is:

1. On a computing device, a method for monitoring performance of a process by a user involving a first object and a second object, the method comprising:
sending a request to a remote server for instructional information for the process;
receiving from the remote server the instructional information, the instructional information including object identification information comprising one or more of three-dimensional object data and an identification function for each of the first object and the second object, and also comprising a set of rules defining one or more portions of the process;
for a selected portion of the process,
receiving image information of a physical scene from a depth camera,
identifying from the image information and the one or more of the three-dimensional object data and the identification function for each of the first object and the second object an operation performed by the user, the operation comprising an interaction of the first object with the second object in the physical scene,
identifying whether one or more of the first object and the second object is a correct object for the operation; and
if one or more of the first object and the second object is an incorrect object for the operation, outputting an alert informing the user of the correct object for the operation;

if the operation does not satisfy a rule associated with the selected portion of the process, outputting an instruction to a display device to instruct how to perform an expected operation to satisfy the rule; and
if the operation does satisfy the rule, advancing to a subsequent portion of the process.

2. The method of claim 1 , wherein the object identification information includes three-dimensional object data usable to identify the first object and the second object in a plurality of different orientations from the image information.

3. The method of claim 1 , wherein the rule specifies one or more of an expected object, an expected first position of the expected object, an expected second position of the expected object, an expected path between the first and second positions, and movement of the expected object between the first and second positions in at least one of six degrees of freedom.

4. The method of claim 3 , further comprising determining that the operation satisfies the rule if the one or more of the expected object, the expected first position of the expected object, the expected second position of the expected object, and the expected path between the first and second positions are met by the operation.

5. The method of claim 3 , further comprising determining that the operation does not satisfy the rule if one or more of the identified object, a beginning position of the identified object, a path of the identified object during the operation, and an ending position of the identified object does not match one or more conditions related to one or more of the expected object, the expected first position, the expected path, and the expected second position of the rule.

6. The method of claim 5 , wherein the rule further specifies a given time threshold in which the expected object is to be moved from the first position to the second position, and further comprising determining that the operation does not satisfy the rule if the identified object is not moved from the beginning position to the ending position by the given time threshold.

7. The method of claim 1 , wherein taking an action based upon whether the operation satisfies the rule comprises, if the operation does satisfy the rule, for a subsequent selected portion of the process:
receiving image information of the physical scene;
identifying from the image information and the object identification information a subsequent operation performed with the interaction of the first object with the second identified object and a different identified object in the physical scene; and
taking an action based upon whether the subsequent operation satisfies a subsequent rule of the set of rules associated with the subsequent selected portion of the process.

8. The method of claim 1 , wherein the instruction comprises one or more of a video instruction, an auditory instruction, and an augmented reality instruction.

9. The method of claim 1 , wherein taking an action based upon whether the operation satisfies the rule comprises storing information indicating whether the operation satisfied the rule.

10. The method of claim 1 , wherein receiving image information further comprises receiving image information from a visible light camera.

11. The method of claim 1 , further comprising tracking movement of the identified object with the received image information in order to identify the operation.

12. On a computing device, a method for monitoring performance of a process involving a first object and a second object, the method comprising:
receiving a set of rules defining one or more actions to be performed by a user on the first object and the second object during the process, the first object and the second object being different than the user;
receiving object identification information regarding the first object and the second object;
for a selected portion of the process,
receiving image information of a physical scene from a depth camera;
identifying from the image information and the object identification information an operation performed by the user on the first object and the second object in the physical scene, the operation comprising an interaction of the first object with the second object;
identifying whether one or more of the first object and the second object is a correct object for the operation;
if one or more of the first object and the second object is an incorrect object for the operation, outputting an alert informing the user of the correct object for the operation;
if the operation does not satisfy a rule, outputting an instruction to a display device to instruct how to perform an expected operation to satisfy the rule; and
if the operation does satisfy the rule, advancing to a subsequent selected portion of the process.

13. The method of claim 12 , wherein advancing to the subsequent selected portion of the process comprises:
receiving image information of the physical scene;
identifying from the image information and the object identification information a subsequent operation performed with a different identified object of the first object and the second object in the physical scene; and
outputting an instruction or advancing to a next selected portion of the process based upon whether the subsequent operation satisfies a subsequent rule of the set of rules associated with the subsequent selected portion of the process.

14. The method of claim 12 , wherein outputting the instruction comprises outputting one or more of an audio instruction and a visual instruction.

15. The method of claim 14 , wherein outputting the visual instruction comprises outputting an augmented reality view of the imaged physical scene to a display device for display, the augmented reality view including a virtual object overlaid on a representation of the physical scene.

16. A hardware storage device holding instructions executable by a logic device to:
send a request to a remote server for instructional information for a process involving a first object and a second object;
receive from the remote server the instructional information, the instructional information including object identification information comprising one or more of three-dimensional object data and an identification function

Assignees

• Microsoft Technology Licensing Llc

Inventors

• White Christopher
• Tsao Anson
• Molyneaux David

Classifications

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06T2207/30108

  Industrial image inspection · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G09B5/06 Primary

  with both visual and audible presentation of the material to be studied · CPC title

• G06Q10/20

  Administration of product repair or maintenance · CPC title