Systems and methods for learning and managing robot user interfaces

Patent metadata

Field	Value
Publication number	US-12084080-B2
Application number	US-202217896187-A
Country	US
Kind code	B2
Filing date	Aug 26, 2022
Priority date	Jul 7, 2022
Publication date	Sep 10, 2024
Grant date	Sep 10, 2024

Abstract

Systems and methods for learning and managing robot user interfaces are disclosed herein. One embodiment generates, based on input data including information about past interactions of a particular user with a robot and with existing HMIs of the robot, a latent space using one or more encoder neural networks, wherein the latent space is a reduced-dimensionality representation of learned behavior and characteristics of the particular user, and uses the latent space as input to train a decoder neural network associated with (1) a new HMI distinct from the existing HMIs or (2) a particular HMI among the existing HMIs to alter operation of the particular HMI. The trained first decoder neural network is deployed in the robot to control, at least in part, operation of the new HMI or the particular HMI in accordance with the learned behavior and characteristics of the particular user.

First claim

What is claimed is:

1. A system for learning and managing robot user interfaces, the system comprising:
a processor; and
a memory storing machine-readable instructions that, when executed by the processor, cause the processor to:
generate, based on input data that includes information about past interactions of a particular user with a robot and with one or more existing Human-Machine Interfaces (HMIs) of the robot, a latent space using one or more encoder neural networks, wherein the latent space is a reduced-dimensionality representation of learned behavior and characteristics of the particular user; and
use the latent space as input to train a first decoder neural network associated with one of (1) a new HMI of the robot distinct from the one or more existing HMIs of the robot and (2) a particular HMI among the one or more existing HMIs of the robot to alter operation of the particular HMI;
wherein the trained first decoder neural network is deployed in the robot to control, at least in part, operation of one of the new HMI and the particular HMI in accordance with the learned behavior and characteristics of the particular user.

2. The system of claim 1 , wherein the robot is a vehicle and the particular user is a driver of the vehicle.

3. The system of claim 2 , wherein the machine-readable instructions to use the latent space as input to train a first decoder neural network associated with a particular HMI among the one or more existing HMIs of the robot to alter operation of the particular HMI include instructions that, when executed by the processor, cause the processor to train the first decoder neural network to perform one of:
tuning a continuous parameter of a safety-related system of the vehicle;
changing a first frequency of activation and an intensity of one or more visual indicators in the vehicle;
changing a second frequency of activation and the volume of one or more audible indicators in the vehicle; and
changing whether the vehicle automatically activates or deactivates a feature in response to detection of a predetermined situation.

4. The system of claim 1 , wherein the robot is a service robot.

5. The system of claim 1 , wherein the machine-readable instructions include further instructions that, when executed by the processor, cause the processor to:
use the latent space as input to train a second decoder neural network to predict future behavior of the particular user; and
use the latent space as input to train a third decoder neural network to label observed behavior of the particular user.

6. The system of claim 1 , wherein the one or more encoder neural networks include a Long Short-Term Memory (LSTM) encoder, an attention network, and a transformer network and the first decoder neural network is one of a fully connected neural network, an LSTM decoder, an attention network, and a reinforcement-learning-based network.

7. The system of claim 1 , wherein the machine-readable instructions to use the latent space as input to train the first decoder neural network include instructions that, when executed by the processor, cause the processor to train the first decoder network to coach the particular user concerning one of the new HMI, the particular HMI, and control of an operational mode of the robot.

8. The system of claim 1 , wherein the information about the past interactions of the particular user with the robot and with the one or more existing HMIs of the robot includes one or more questionnaire responses from the particular user.

9. A non-transitory computer-readable medium for learning and managing robot user interfaces and storing instructions that, when executed by a processor, cause the processor to:
generate, based on input data that includes information about past interactions of a particular user with a robot and with one or more existing Human-Machine Interfaces (HMIs) of the robot, a latent space using one or more encoder neural networks, wherein the latent space is a reduced-dimensionality representation of learned behavior and characteristics of the particular user; and
use the latent space as input to train a first decoder neural network associated with one of (1) a new HMI of the robot distinct from the one or more existing HMIs of the robot and (2) a particular HMI among the one or more existing HMIs of the robot to alter operation of the particular HMI;
wherein the trained first decoder neural network is deployed in the robot to control, at least in part, operation of one of the new HMI and the particular HMI in accordance with the learned behavior and characteristics of the particular user.

10. The non-transitory computer-readable medium of claim 9 , wherein the robot is a vehicle and the particular user is a driver of the vehicle.

11. The non-transitory computer-readable medium of claim 10 , wherein the instructions to use the latent space as input to train a first decoder neural network associated with a particular HMI among the one or more existing HMIs of the robot to alter operation of the particular HMI include instructions that, when executed by the processor, cause the processor to train the first decoder neural network to perform one of:
tuning a continuous parameter of a safety-related system of the vehicle;
changing a first frequency of activation and an intensity of one or more visual indicators in the vehicle;
changing a second frequency of activation and the volume of one or more audible indicators in the vehicle; and
changing whether the vehicle automatically activates or deactivates a feature in response to detection of a predetermined situation.

12. The non-transitory computer-readable medium of claim 9 , wherein the instructions include further instructions that, when executed by the processor, cause the processor to:
use the latent space as input to train a second decoder neural network to predict future behavior of the particular user; and
use the latent space as input to train a third decoder neural network to label observed behavior of the particular user.

13. A method, comprising:
generating, based on input data that includes information about past interactions of a particular user with a robot and with one or more existing Human-Machine Interfaces (HMIs) of the robot, a latent space using one or more encoder neural networks, wherein the latent space is a reduced-dimensionality representation of learned behavior and characteristics of the particular user; and
using the latent space as input to train a first decoder neural network associated with one of (1) a new HMI of the robot distinct from the one or more existing HMIs of the robot and (2) a particular HMI among the one or more existing HMIs of the robot to alter operation of the particular HMI;
wherein the trained first decoder neural network is deployed in the robot to control, at least in part, operation of one of the new HMI and the particular HMI in accordance with the learned behavior and characteristics of the particular user.

14. The method of claim 13 , wherein the robot is a vehicle and the particular user is a driver of the vehicle.

15. The method of claim 14 , wherein the using the latent space as input to train a first decoder neural network associated with a particular HMI among the one or more existing HMIs of the robot to alter operation of the particular HMI includes training the first decoder neural network to perform one of:
tuning a continuous parameter of a safety-related system of the vehicle;
changing a first frequency of activation and an intensity of one or more visual indicators in the vehicle;
changing a second frequency of activation and the v

Assignees

• Toyota Res Inst Inc
• Toyota Motor Co Ltd

Inventors

• Rosman Guy
• Brooks Daniel J
• Stent Simon A I
• Chen Tiffany
• Sumner Emily Sarah
• Hakimi Shabnam
• Decastro Jonathan
• Gopinath Deepak Edakkattil

Classifications

• B25J9/161

  Hardware, e.g. neural networks, fuzzy logic, interfaces, processor · CPC title

• B60W2050/146

  Display means · CPC title

• B60W2050/143

  Alarm means · CPC title

• B60W2050/0083

  Setting, resetting, calibration · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title