Efficient robot control based on inputs from remote client devices

Patent metadata

Field	Value
Publication number	US-2023398690-A1
Application number	US-202318233261-A
Country	US
Kind code	A1
Filing date	Aug 11, 2023
Priority date	Jul 26, 2019
Publication date	Dec 14, 2023
Grant date	—

Abstract

Utilization of user interface inputs, from remote client devices, in controlling robot(s) in an environment. Implementations relate to generating training instances based on object manipulation parameters, defined by instances of user interface input(s), and training machine learning model(s) to predict the object manipulation parameter(s). Those implementations can subsequently utilize the trained machine learning model(s) to reduce a quantity of instances that input(s) from remote client device(s) are solicited in performing a given set of robotic manipulations and/or to reduce the extent of input(s) from remote client device(s) in performing a given set of robotic operations. Implementations are additionally or alternatively related to mitigating idle time of robot(s) through the utilization of vision data that captures object(s), to be manipulated by a robot, prior to the object(s) being transported to a robot workspace within which the robot can reach and manipulate the object.

First claim

What is claimed is:

1 . A method, comprising:
receiving, from one or more vision components in an environment, vision data that captures features of the environment, including object features of an object that is located in the environment; generating, based on processing the vision data using a machine learning model:
a predicted parameter for use in controlling the robot in the environment, and
a confidence measure for the predicted parameter;
determining whether the confidence measure, for the predicted parameter, satisfies a threshold confidence measure; responsive to determining the confidence measure fails to satisfy the threshold confidence measure:
transmitting, via one or more networks to a remote client device, an object representation of the object that is generated based on the object features and a visual indication of the predicted parameter,
wherein transmitting, to the remote client device, the visual indication of the predicted parameter, is further responsive to determining that the confidence measure satisfies a lower bound threshold confidence measure that is less indicative of confidence than the threshold confidence measure;

receiving, via the one or more networks and from the remote client device, data that is generated based on one or more user interface inputs, at the remote client device, responsive to rendering the object representation and the visual indication at the remote client device; and
causing the robot to be controlled in the environment in dependence on the received data; and
responsive to determining the confidence measure satisfies the threshold confidence measure:
causing the robot to be controlled in the environment in accordance with the predicted parameter, and without transmitting the visual indication to any remote client device for confirmation prior to the robot being controlled in accordance with the predicted parameter.

2 . The method of claim 1 , further comprising:
responsive to determining that the confidence measure fails to satisfy the lower bound threshold confidence measure:
transmitting, to the remote client device, the object representation of the object without transmitting any visual indication of the predicted parameter.

3 . The method of claim 1 , wherein the received data indicates a confirmation of the predicted parameter, and wherein causing the robot to be controlled in the environment in dependence on the received data comprises causing the robot to be controlled in the environment in accordance with the predicted parameter, responsive to the received data indicating the confirmation of the predicted parameter.

4 . The method of claim 3 , wherein a confirmation user interface element is rendered at the remote client device along with rendering of the object representation and the visual indication at the remote client device, and wherein the received data indicates the confirmation of the predicted parameter in response to user interface input being directed to the confirmation user interface element at the remote client device.

5 . The method of claim 1 , wherein the received data indicates an alternative parameter, and wherein causing the robot to be controlled in the environment in dependence on the received data comprises causing the robot to be controlled in the environment in accordance with the alternative parameter, responsive to the received data indicating the alternative parameter.

6 . The method of claim 5 , wherein an alternate parameter user interface element is rendered at the remote client device along with rendering of the object representation and the visual indication at the remote client device, and wherein the received data indicates the alternative parameter in response to user interface input being directed to the alternate parameter user interface element at the remote client device.

7 . The method of claim 5 , further comprising:
generating a positive training instance based on the alternative parameter indicated by the received data; and training the machine learning model based on the positive training instance.

8 . The method of claim 1 , wherein both a confirmation user interface element and an alternate parameter user interface element are rendered at the remote client device along with rendering of the object representation and the visual indication at the remote client device.

9 . A method, comprising:
receiving, from one or more vision components in an environment, vision data that captures features of the environment, including object features of an object that is located in the environment; generating, based on processing the vision data using a machine learning model:
a predicted parameter for use in controlling the robot in the environment, and
a confidence measure for the predicted parameter;
determining whether the confidence measure, for the predicted parameter, satisfies a threshold confidence measure;
generating a visual representation to transmit via one or more networks to a remote client device, generating the visual representation comprising:
including, in the visual representation, an object representation of the object that is generated based on the object features;
determining whether to include, in the visual representation, a visual indication of the predicted parameter, based on whether the confidence measure satisfies a lower bound threshold confidence measure;
transmitting the visual representation to the remote client device; receiving, via the one or more networks and from the remote client device, data that is generated based on one or more user interface inputs, at the remote client device, responsive to rendering the visual representation at the remote client device; and causing a robot to be controlled in the environment in dependence on the received data.

10 . The method of claim 9 , wherein the visual representation lacks any visual indication of the predicted parameter based on determining that the confidence measure fails to satisfy the lower bound threshold confidence measure.

11 . The method of claim 9 , wherein the received data indicates a confirmation of the predicted parameter, and wherein causing the robot to be controlled in the environment in dependence on the received data comprises causing the robot to be controlled in the environment in accordance with the predicted parameter, responsive to the received data indicating the confirmation of the predicted parameter.

12 . The method of claim 11 , wherein a confirmation user interface element is rendered at the remote client device along with rendering of the object representation and the visual indication at the remote client device, and wherein the received data indicates the confirmation of the predicted parameter in response to user interface input being directed to the confirmation user interface element at the remote client device.

13 . The method of claim 9 , wherein the received data indicates an alternative parameter, and wherein causing the robot to be controlled in the environment in dependence on the received data comprises causing the robot to be controlled in the environment in accordance with the alternative parameter, responsive to the received data indicating the alternative parameter.

14 . The method of claim 13 , wherein an alternate parameter user interface element is rendered at the remote client device along with rendering of the object representation and the visual indication at the remote client device, and wherein the received data indicates the alternative parameter in response to user interface input being directed to the alternate parameter user interface element

Assignees

• Google Llc

Inventors

• Lee Johnny
• Welker Stefan

Classifications

• B25J9/1689 Primary

  Teleoperation · CPC title

• B25J9/163 Primary

  learning, adaptive, model based, rule based expert control · CPC title

• B25J9/1697

  Vision controlled systems · CPC title

• B25J9/1612

  characterised by the hand, wrist, grip control · CPC title

• G05B2219/36422

  During teaching shut off, disable motor to move arm easy · CPC title