Anticipating user and object poses through task-based extrapolation for robot-human collision avoidance

Patent metadata

Field	Value
Publication number	US-11945117-B2
Application number	US-202117198128-A
Country	US
Kind code	B2
Filing date	Mar 10, 2021
Priority date	Mar 10, 2021
Publication date	Apr 2, 2024
Grant date	Apr 2, 2024

Abstract

An example method includes determining objects and actions associated with the objects for completing a task to be executed by a robotic system, where each action is associated with trajectory. The method further includes determining a pose for each person in an environment associated with the robotic system, predicting a trajectory for each person based on the determined pose associated with the respective person and the actions and trajectories associated with the actions, and adjusting trajectories for one or more of the actions to be performed by the robotic system based on the predicted trajectories for each person.

First claim

What is claimed is:

1. A method comprising, by a robotic system:
determining, for a task to be executed at least in part by the robotic system, one or more objects and one or more actions associated with the one or more objects for completing the task, wherein each action is associated with a trajectory;
determining, for each of one or more people in an environment associated with the robotic system, a pose of the respective person;
predicting, for each of the one or more people and based on the determined pose associated with the respective person, a three-dimensional, bounded trajectory volume comprising a plurality of 3D volumes for the respective person, wherein each 3D volume of the predicted trajectory volume for each person is associated with a predicted probability that the respective person or an action of the respective person will occupy that 3D volume during a particular timeframe; and
determining whether one or more trajectories associated with one or more actions of the robotic system passes through the trajectory volume during the particular timeframe; and
in response to a determination that one or more trajectories associated with one or more actions of the robotic system passes through the trajectory volume during the particular timeframe, then adjusting the one or more trajectories associated with one or more actions of the robotic system; and
in response to a determination that one or more trajectories associated with one or more actions of the robotic system does not pass through the trajectory volume in the particular timeframe, then leaving the one or more trajectories associated with one or more actions of the robotic system unchanged.

2. The method of claim 1 , further comprising:
determining, for each of the one or more objects, a pose and one or more attributes of the respective object based on sensor data captured by one or more sensors associated with the robotic system.

3. The method of claim 2 , wherein the three-dimensional trajectory volume is further predicted based on the pose and or more attributes of at least one of the one or more objects.

4. The method of claim 1 , further comprising:
predicting, for each of the one or more people, a pose of a hand of the respective person based on one or more of:
attributes of one or more of the objects;
poses of one or more of the objects;
one or more of the actions;
human kinematics associated with the respective person;
a three-dimensional (3D) scene representation of the environment associated with the robotic system, wherein the 3D scene representation is generated based on sensor data captured by one or more sensors associated with the robotic system; or
the determined pose associated with the respective person,

wherein predicting the three-dimensional trajectory volume for each of the one or more people is further based on the predicted pose of the hand of the respective person.

5. The method of claim 1 , wherein each 3D volume comprises a voxel.

6. The method of claim 1 , wherein adjusting the one or more trajectories associated with one or more actions of the robotic system comprises one or more of:
adjusting a route associated with at least one of the one or more trajectories;
adjusting a speed associated with at least one of the one or more trajectories;
suspending at least one of the one or more trajectories;
changing at least one of the one or more trajectories to a trajectory associated with another task; or
alerting the one or more people about at least one of the one or more trajectories.

7. The method of claim 1 , further comprising:
generating a scene graph comprising a plurality of nodes, wherein each node represents one or more of:
an object associated with one or more attributes;
a pose of one of the one or more people;
an action by one of the one or more people; or
an action of the one or more actions to be performed by the robotic system, and

wherein generating the scene graph is based on one or more of:
an analysis of each of a plurality of tasks based on natural language understanding;
human pose estimation of poses associated with the one or more people based on sensor data captured by one or more sensors associated with the robotic system; or
object detection based on the sensor data.

8. A robotic system comprising:
a computing system with control software;
a robot controller;
one or more robotic limbs;
one or more non-transitory computer-readable storage media including instructions; and
one or more processors coupled to the storage media, the one or more processors configured to execute the instructions to:
determine, for a task to be executed at least in part by the robotic system, one or more objects and one or more actions associated with the one or more objects for completing the task, wherein each action is associated with a trajectory;
determine, for each of one or more people in an environment associated with the robotic system, a pose of the respective person;
predict, for each of the one or more people and based on the determined pose associated with the respective person, a three-dimensional, bounded trajectory volume comprising a plurality of 3D volumes for the respective person, wherein each 3D volume of the predicted trajectory volume for each person is associated with a predicted probability that the respective person or an action of the respective person will occupy that 3D volume during a particular timeframe; and
determine whether one or more trajectories associated with one or more actions of the robotic system passes through the trajectory volume during the particular timeframe; and
in response to a determination that one or more trajectories associated with one or more actions of the robotic system passes through the trajectory volume during the particular timeframe, then adjust the one or more trajectories associated with one or more actions of the robotic system; and
in response to a determination that one or more trajectories associated with one or more actions of the robotic system does not pass through the trajectory volume in the particular timeframe, then leave the one or more trajectories associated with one or more actions of the robotic system unchanged.

9. The robotic system of claim 8 , wherein the processors are further configured to execute the instructions to:
determine, for each of the one or more objects, a pose and one or more attributes of the respective object based on sensor data captured by one or more sensors associated with the robotic system.

10. The robotic system of claim 9 , wherein the three-dimensional trajectory volume is further predicted based on the pose and or more attributes of at least one of the one or more objects.

11. The robotic system of claim 8 , wherein the processors are further configured to execute the instructions to:
predict, for each of the one or more people, a pose of a hand of the respective person based on one or more of:
attributes of one or more of the objects;
poses of one or more of the objects;
one or more of the actions;
human kinematics associated with the respective person;
a three-dimensional (3D) scene representation of the environment associated with the robotic system, wherein the 3D scene representation is generated based on sensor data captured by one or more sensors associated with the robotic system; or
the determined pose associated with the respective person,

wherein predicting the three-dimensional trajectory volume for each of the one or more people is further based on the predicted pose of the hand of the respective person.

12. The robotic system of claim 8 , wherein each 3D volume comprises a voxel.

Assignees

• Samsung Electronics Co Ltd
• Samsung Electronics Co Ltd

Inventors

• Schoessler Philipp
• Harms Brian
• Hajash Kathleen Sofia
• Mason Dane

Classifications

• B25J9/1666 Primary

  Avoiding collision or forbidden zones · CPC title

• B25J9/163

  learning, adaptive, model based, rule based expert control · CPC title

• B25J9/1651

  acceleration, rate control · CPC title

• B25J9/1661

  characterised by task planning, object-oriented languages · CPC title

• B25J13/089

  Determining the position of the robot with reference to its environment · CPC title