Method for tele-robotic operations over time-delayed communication links

Patent metadata

Field	Value
Publication number	US-9776325-B1
Application number	US-201314073624-A
Country	US
Kind code	B1
Filing date	Nov 6, 2013
Priority date	Mar 13, 2013
Publication date	Oct 3, 2017
Grant date	Oct 3, 2017

Abstract

Described is system for tele-robotic operations over time-delayed communication links. Sensor data is acquired from at least one sensor for sensing surroundings of a robot having at least one robotic arm for manipulating an object. A three-dimensional model of the sensed surroundings is generated, and the sensor data is fit to the three-dimensional model. Using the three-dimensional model, a user demonstrates a movement path for the at least one robotic arm. A flow field representing the movement path is generated and combined with obstacle-repellent forces to provide force feedback to the user through a haptic device. The flow field comprises a set of parameters, and the set of parameters are transmitted to the robot to execute a movement of the at least one robotic arm for manipulating the object.

First claim

What is claimed is:

1. A system for tele-robotic operations, the system comprising:
one or more processors and a non-transitory memory having instructions encoded thereon such that when the instructions are executed, the one or more processors perform operations of:
acquiring sensor data from at least one sensor for sensing surroundings of a robot having at least one robotic arm for manipulating an object;
generating a three-dimensional model of the sensed surroundings and fitting the sensor data to the three-dimensional model;
using the three-dimensional model to generate a movement path for the at least one robotic arm;
generating a flow field representing the movement path;
using the flow field in a simulation to provide force feedback to a user through a haptic device, wherein the flow field comprises a set of parameters; and
following the simulation, transmitting the set of parameters used in the simulation to the robot, wherein the set of parameters define the floss field, and wherein the flow field is used to execute a movement of the at least one robotic arm for manipulating the object,
wherein the flow field is updated in real-time according to the sensor data.

2. The system as set forth in claim 1 , wherein the flow field changes over time during execution of the movement of the at least one robotic arm.

3. The system as set forth in claim 2 , wherein the one or more processors further perform an operation of continuously updating a position and as orientation of the three-dimensional model based on incoming three-dimensional point cloud data generated from the sensor data.

4. The system as set forth in claim 3 , wherein the one or more processors further perform an operation of updating an orientation of the flow field according to updates in the three-dimensional model's position and orientation based on data received from the robot.

5. The system as set forth in claim 1 , wherein the one or more processors further perform an operation of enabling a user to label at least one obstacle in the three-dimensional model, wherein the at least one obstacle is augmented with at least one repellent force field having a strength that the user can regulate, and wherein the haptic device provides a force feedback having a strength that is proportional to a value representing the at least one repellent force field.

6. The system as set forth in claim 5 , wherein the one or more processors further perform an operation of:
enabling the user to practice the movement path through the flow field prior to transmitting the set of parameters to the robot; and
if a problem occurs, then enabling the user to:
modify the at least one repellent force field;
re-demonstrate the movement path; or
edit the movement path.

7. A computer-implemented method for tele-robotic operations, comprising:
an act of causing a data processor to execute instructions stored on a non-transitory memory such that upon execution, the data processor performs operations of:
acquiring sensor data from at least one sensor for sensing surroundings of a robot having at least one robotic arm for manipulating an object;
generating a three-dimensional model of the sensed surroundings and fitting the sensor data to the three-dimensional model;
using the three-dimensional model to generate a movement path for the at least one robotic arm;
generating a flow field representing the movement path;
using the flow field in a simulation to provide force feedback to a user through a haptic device, wherein the flow field comprises a set of parameters; and
following the simulation, transmitting the set of parameters used in the simulation to the robot, wherein the set of parameters define the flow field, and wherein the flow field is used to execute a movement of the at least one robotic arm for manipulating the object,
wherein the flow field is updated in real-time according to the sensor data.

8. The method as set forth in claim 7 , wherein the flow field changes over time during execution of the movement of the at least one robotic arm.

9. The method as set forth in claim 8 , wherein the data processor further performs an operation of continuously updating a position and an orientation of the three-dimensional model based on incoming three-dimensional point cloud data generated from the sensor data.

10. The method as set forth in claim 9 , wherein the data processor further performs an operation of updating an orientation of the flow field according to updates in the three-dimensional model's position and orientation based on data received from the robot.

11. The method as set forth in claim 7 , wherein the data processor further performs an operation of enabling a user to label at least one obstacle in the three-dimensional model, wherein the at least one obstacle is augmented with at least one repellent force field having a strength that the user can regulate, and wherein the haptic device provides a force feedback having a strength that is proportional to a value representing the at least one repellent force field.

12. The method as set forth in claim 11 , wherein the data processor further performs operations of:
enabling the user to practice the movement path through the flow field prior to transmitting the set of parameters to the robot; and
if a problem occurs, then enabling the user to:
modify the at least one repellent force field;
re-demonstrate the movement path; or
edit the movement path.

13. A computer program product for tele-robotic operations, the computer program product comprising computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of:
acquiring sensor data from at least one sensor for sensing surroundings of a robot having at least one robotic arm for manipulating an object;
generating a three-dimensional model of the sensed surroundings and fitting the sensor data to the three-dimensional model;
using the three-dimensional model to generate a movement path for the at least one robotic arm;
generating a flow field representing the movement path;
using the flow field in a simulation to provide force feedback to a user through a haptic device, wherein the flow field comprises a set of parameters; and
following the simulation, transmitting the set of parameters used in the simulation to the robot, wherein the set of parameters define the flow field, and wherein the flow field is used to execute a movement of the at least one robotic arm for manipulating the object,
wherein the flow field is updated in real-time according to the sensor data.

14. The computer program product as set forth in claim 13 , wherein the flow field changes over time during execution of the movement of the at least one robotic arm.

15. The computer program product as set forth in claim 14 , further comprising instructions for causing the processor to perform an operation of continuously updating a position and an orientation of the three-dimensional model based on incoming three-dimensional point cloud data generated from the sensor data.

16. The computer program product as set forth in claim 15 , further comprising instructions for causing the processor to perform an operation of updating an orientation of the flow field according to updates in the three-dimensional model's position and orientation based on data received from the robot.

17. The computer program product as set forth in claim 13 , further comprising instructions for causing the processor to perform an opera

Assignees

• Hrl Lab Llc

Inventors

• Hoffmann Heiko
• Payton David W
• De Sapio Vincent

Classifications

• B25J9/1671 Primary

  characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems · CPC title

• G05B2219/40122

  Manipulate virtual object, for trajectory planning of real object, haptic display · CPC title

• B25J9/163

  learning, adaptive, model based, rule based expert control · CPC title

• G05B2219/40124

  During manipulator motion, sensor feedback to adapt model in memory · CPC title

• B25J9/1605

  Simulation of manipulator lay-out, design, modelling of manipulator · CPC title