Efficient robot control based on inputs from remote client devices

Patent metadata

Field	Value
Publication number	US-11724398-B2
Application number	US-202117535393-A
Country	US
Kind code	B2
Filing date	Nov 24, 2021
Priority date	Jul 26, 2019
Publication date	Aug 15, 2023
Grant date	Aug 15, 2023

Abstract

Utilization of user interface inputs, from remote client devices, in controlling robot(s) in an environment. Implementations relate to generating training instances based on object manipulation parameters, defined by instances of user interface input(s), and training machine learning model(s) to predict the object manipulation parameter(s). Those implementations can subsequently utilize the trained machine learning model(s) to reduce a quantity of instances that input(s) from remote client device(s) are solicited in performing a given set of robotic manipulations and/or to reduce the extent of input(s) from remote client device(s) in performing a given set of robotic operations. Implementations are additionally or alternatively related to mitigating idle time of robot(s) through the utilization of vision data that captures object(s), to be manipulated by a robot, prior to the object(s) being transported to a robot workspace within which the robot can reach and manipulate the object.

First claim

What is claimed is:

1. A method comprising:
receiving, from one or more vision components in a first area of an environment, vision data that captures features of the first area at a first time, including object features of an object that is located in the first area at the first time;
prior to completion of transporting of the object from the first area to a disparate robot workspace, of the environment, that is not captured by the vision data:
transmitting, via one or more networks to a remote client device, a visual representation that is generated based on the vision data that captures the features of the first area at the first time, wherein the visual representation includes an object representation generated based on the object features,
wherein the object representation includes a bounding shape defined by:
two or more coordinates that define a center of the bounding shape,
one or more orientation values that define an orientation of the bounding shape, and
at least one size value that defines a size of the bounding shape;

wherein the visual representation omits one or more other objects, of the first area, that are visible in the vision data; and

receiving, via the one or more networks and from the remote client device, data that is generated based on one or more user interface inputs, at the remote client device, that are directed at the visual representation, that is generated based on the vision data that captures the features of the first area at the first time, when the visual representation is rendered at the remote client device;

determining, based on the data, one or more object manipulation parameters for manipulating of the object by a robot operating in the robot workspace; and
causing the robot to manipulate the object, in accordance with the one or more object manipulation parameters,
wherein the robot manipulates the object in accordance with the one or more object manipulation parameters responsive to the robot detecting, using one or more robot sensors, that the object is within the robot workspace,
wherein the one or more robot sensors are in addition to the vision components in the first area of the environment, and
wherein the object is within the robot workspace after transporting of the object from the first area to the robot workspace and at a second time that is subsequent to the first time.

2. The method of claim 1 , wherein determining the one or more object manipulation parameters is also prior to completion of transporting of the object from the first area to the robot workspace.

3. The method of claim 1 , wherein the one or more object manipulation parameters include a grasp pose for grasping of the object, and wherein causing the robot to manipulate the object, in accordance with the one or more object manipulation parameters, comprises:
causing an end effector of the robot to traverse to the grasp pose, and attempt a grasp of the object after traversing to the grasp pose.

4. The method of claim 1 , wherein the data defines one or more poses and/or one or more points relative to a first reference frame, and wherein generating the one or more object manipulation parameters comprises:
transforming the one or more poses and/or the one or more points to a robot frame that is disparate from the reference frame; and
using the transformed poses and/or points in generating the object manipulation parameters.

5. The method of claim 1 , further comprising:
subsequent to causing the robot to manipulate the object:
determining, based on additional sensor data from one or more additional sensors, a measure of success of the manipulation;
generating a positive training instance based on the measure of success satisfying a threshold; and
training a machine learning model based on the positive training instance.

6. The method of claim 5 , wherein the one or more additional sensors include at least one of the one or more robot sensors or a weight sensor in the environment.

7. The method of claim 5 , wherein generating the positive training instance comprises generating training instance input, of the positive training instance, based on the vision data or based on robot vision data from one or more robot vision components of the robot.

8. The method of claim 7 , wherein generating the positive training instance comprises generating training instance output, of the positive training instance, based on the object manipulation parameters.

9. The method of claim 8 , further comprising:
subsequent to training the machine learning model based on the positive training instance, further comprising:
processing, using the machine learning model, additional vision data that captures an additional object;
generating, based on the processing, one or more predicted object manipulation parameters for the additional object; and
causing the robot to manipulate the additional object in accordance with the one or more predicted object manipulation parameters.

10. The method of claim 9 , further comprising:
transmitting, to the remote client device or to an additional remote client device, a visual indication of the predicted object manipulation parameters;
receiving, from the remote client device or the additional remote client device, an indication that affirmative user interface input was received responsive to presentation of the visual indication of the predicted object manipulation parameters;
wherein causing the robot to manipulate the additional object in accordance with the one or more predicted object manipulation parameters is responsive to receiving the indication that affirmative user interface input was received.

11. The method of claim 10 , further comprising:
generating, based on the processing, a confidence measure for the one or more predicted object manipulation parameters;
wherein transmitting the visual indication of the predicted object manipulation parameters is responsive to the confidence measure failing to satisfy a threshold confidence measure.

12. The method of claim 8 , further comprising:
subsequent to training the machine learning model based on the positive training instance, further comprising:
processing, using the machine learning model, additional vision data that captures an additional object;
generating, based on the processing, one or more predicted object manipulation parameters for the additional object;
transmitting, to the remote client device or to an additional remote client device, a visual indication of the predicted object manipulation parameters;
receiving, from the remote client device or the additional remote client device, an indication of alternate object manipulation parameters defined via user interface input received responsive to presentation of the visual indication of the predicted object manipulation parameters; and
causing, responsive to receiving the alternate object manipulation parameters, the robot to manipulate the additional object in accordance with the one or more alternate object manipulation parameters.

13. A method, comprising:
receiving, from one or more vision components in an environment, vision data that captures features of the environment, including object features of an object that is located in the environment;
generating, based on processing the vision data using a machine learning model:
a predicted object manipulation parameter for the object, the predicted object manipulation parameter being a predicted grasp pose for grasping the object or a waypoint to encounter in traversing a robot to interact with the object, and
a confidence measure for the predicted object manipulation parameter;

determining whether the

Assignees

• Google Llc

Inventors

• Lee Johnny
• Welker Stefan

Classifications

• B25J9/1689 Primary

  Teleoperation · CPC title

• B25J9/163 Primary

  learning, adaptive, model based, rule based expert control · CPC title

• B25J9/1697

  Vision controlled systems · CPC title

• B25J9/1612

  characterised by the hand, wrist, grip control · CPC title

• G05B2219/36422

  During teaching shut off, disable motor to move arm easy · CPC title