System and method for embodied authoring of human-robot collaborative tasks with augmented reality

Patent metadata

Field	Value
Publication number	US-12145267-B2
Application number	US-202017022216-A
Country	US
Kind code	B2
Filing date	Sep 16, 2020
Priority date	Sep 18, 2019
Publication date	Nov 19, 2024
Grant date	Nov 19, 2024

Abstract

A system and method for authoring and performing Human-Robot-Collaborative (HRC) tasks is disclosed. The system and method adopt an embodied authoring approach in Augmented Reality (AR), for spatially editing the actions and programming the robots through demonstrative role-playing. The system and method utilize an intuitive workflow that externalizes user's authoring as demonstrative and editable AR ghost, allowing for spatially situated visual referencing, realistic animated simulation, and collaborative action guidance. The system and method utilize a dynamic time warping (DTW) based collaboration model which takes the real-time captured motion as inputs, maps it to the previously authored human actions, and outputs the corresponding robot actions to achieve adaptive collaboration.

First claim

What is claimed is:

1. A method for authoring a human-robot collaborative task in which a robot collaborates with a human, the method comprising:
recording, with at least one sensor, during a first time period, human motions of a human as the human demonstrates the human-robot collaborative task in an environment, the recorded human motions including a plurality of recorded positions of the human in the environment over a period of time;
displaying, on a display, during a second time period that is subsequent to the first time period, an augmented reality graphical user interface including a graphical representation of the recorded human motions that is superimposed on the environment such that the graphical representation appears within the environment at the plurality of recorded positions of the human in the environment;
displaying, in the graphical user interface on the display, during the second time period while the graphical representation of the recorded human motions is displayed, a virtual representation of the robot that is superimposed on the environment and which can be manipulated by the human by providing user inputs;
receiving, via a user interface, during the second time period, user inputs defining manipulations of the virtual representation of the robot, the manipulations being graphically represented by the virtual representation of the robot;
determining, with a processor, during the second time period, a sequence of robot motions to be performed by a robot in concert with a performance of human motions that match the recorded human motions, based on the manipulations of the virtual representation of the robot;
storing, in a memory, during the second time period, the recorded human motions and the sequence of robot motions to be performed by the robot;
detecting, during a third time period that is subsequent to the second time period, the performance of human motions that match the recorded human motions by one of the human and a further human, the detecting including (i) recording, with the at least one sensor, a real-time position of the one of the human and the further human and (ii) comparing, with the processor, the real-time position to the plurality of recorded positions of the human in the recorded human motions; and
generating, with the processor, and transmitting to the robot, with a transceiver, during the third time period, a plurality of commands configured to operate the robot to perform the sequence of robot motions in concert with the performance of the human motions that match the recorded human motions.

2. The method of claim 1 further comprising:
during the performance of human motions that match the recorded human motions by the one of (i) the human and (ii) the further human:
displaying, in the graphical user interface on the display, a virtual representation of a portion of the recorded human motions that have not yet been performed by the one of (i) the human and (ii) the further human, which is superimposed on the environment based on the plurality of recorded positions of the human in the environment.

3. The method of claim 1 , the recording the human motions further comprising:
displaying, in the graphical user interface on the display, a virtual representation of an object that is superimposed on the environment and which can be manipulated by the human by providing user inputs; and
recording, with the at least one sensor, the human motions of the human as the human demonstrates the human-robot collaborative task in the environment, the recorded human motions including the plurality of recorded positions of the human in the environment over the period of time and manipulations of the virtual representation of the object that are performed by the human by providing user inputs during the period of time.

4. The method of claim 1 , the determining the sequence of robot motions to be performed by the robot further comprising:
displaying, in the graphical user interface on the display, a virtual representation of an object, which is superimposed on the environment and which can be manipulated by the virtual representation of the robot by the human by providing user inputs,
wherein the sequence of robot motions include manipulations to be performed of a real-world object corresponding to the virtual representation of the object.

5. The method of claim 1 further comprising:
grouping, with a processor, respective subsets of the recorded human motions into a plurality of motion groups based on user inputs received from the human; and
determining, with the processor, based on user inputs received from the human, for each respective motion group of the plurality of motion groups, a respective set of robot motions to be performed by the robot in concert with a performance of human motions that match the respective motion group,
wherein the sequence of robot motions to be performed by the robot is formed by the set of robot motions to be performed by the robot for each of the plurality of motion groups.

6. The method of claim 5 further comprising:
receiving, for each respective motion group of the plurality of motion groups, a user input from the human designating the respective motion group as one of:
(i) a first type in which the corresponding set of robot motions is to be performed by the robot in synchronization with a progress of a performance of human motions that match the respective motion group; and
(ii) a second type in which the corresponding set of robot motions is to be performed by the robot responsive to a completion of a performance of human motions that match the respective motion group.

7. The method of claim 6 further comprising:
detecting, with the at least one sensor, a plurality of real-time positions of one of (i) the human and (ii) a further human during a performance of human motions that match a respective motion group of the plurality of motion groups that is designated as the first type;
mapping, with the processor, the plurality of real-time positions to the respective subset of recorded human motions of the respective motion group; and
based on the mapping of the plurality of real-time positions, generating, with the processor, and transmitting to the robot, with the transceiver, at least one command configured to operate the robot to perform the respective set of robot motions that match the respective motion group in synchronization with a progress of the performance of human motions that match the respective motion group.

8. The method of claim 6 further comprising:
detecting, with the at least one sensor, a plurality of real-time positions of one of (i) the human and (ii) a further human during a performance of human motions that match a respective motion group of the plurality of motion groups that is designated as the second type;
mapping, with the processor, the plurality of real-time positions to the respective subset of recorded human motions of the respective motion group; and
based on the mapping of the plurality of real-time positions, generating, with the processor, and transmitting to the robot, with a transceiver, at least one command configured to operate the robot to perform the respective set of robot motions that match the respective motion group responsive to a completion of the performance of human motions that match the respective motion group.

9. The method of claim 1 , the displaying the graphical user interface further comprising:
displaying, in the graphical user interface on the display, a plurality of graphical representations of the human, each superimposed on the environment at a respective one of the plurality of recorded positions of the human in the environment.

10. The method of clai

Assignees

• Purdue Research Foundation

Inventors

• Ramani Karthik
• Huo Ke
• Cao Yuanzhi
• Wang Tianyi

Classifications

• G05B2219/40202

  Human robot coexistence · CPC title

• G05B2219/36442

  Automatically teaching, teach by showing · CPC title

• G05B2219/35444

  Gesture interface, controlled machine observes operator, executes commands · CPC title

• G05B2219/39451

  Augmented reality for robot programming · CPC title

• B25J9/1605 Primary

  Simulation of manipulator lay-out, design, modelling of manipulator · CPC title