Facilitating robotic control using a virtual reality interface

Patent metadata

Field	Value
Publication number	US-11625030-B2
Application number	US-201816237035-A
Country	US
Kind code	B2
Filing date	Dec 31, 2018
Priority date	Feb 23, 2015
Publication date	Apr 11, 2023
Grant date	Apr 11, 2023

Abstract

A method of deriving autonomous control information involves receiving one or more sets of associated environment sensor information and device control instructions. Each set of associated environment sensor information and device control instructions includes environment sensor information representing an environment associated with an operator controllable device and associated device control instructions configured to cause the operator controllable device to simulate at least one action taken by at least one operator experiencing a representation of the environment generated from the environment sensor information. The method also involves deriving autonomous control information from the one or more sets of associated environment sensor information and device control instructions, the autonomous control information configured to facilitate generating autonomous device control signals from autonomous environment sensor information representing an environment associated with an autonomous device, the autonomous device control signals configured to cause the autonomous device to take at least one autonomous action.

First claim

What is claimed is:

1. A method for training an autonomous device, the method comprising:
capturing, using a sensor, a first data corresponding to a first action of the autonomous device;
transmitting the first data to a virtual reality interface that is viewable by an operator, wherein the first data is displayed to the operator by using the virtual reality interface, wherein the first data is labeled as representing a failed task by the operator if the first action is not successful;
controlling the autonomous device, via an input control device by the operator, to cause the autonomous device to undertake a second action, the second action associated with a second data,
wherein if the first data is not labeled as representing a failed task, the first data and the second data are utilized by an artificial intelligence model for future control of the autonomous device; and
generating a control signal by the artificial intelligence model.

2. The method of claim 1 , further comprising controlling the autonomous device to undertake a third action, wherein the third action is determined by analysis of the first data and the second data by the artificial intelligence model.

3. The method of claim 1 , wherein the first data and the second data are captured using a camera.

4. The method of claim 1 , wherein the artificial intelligence model is a deep learning model.

5. The method of claim 1 , wherein the input control device comprises at least one of a joystick, a keyboard, a mouse, or a game pad that is coupled to the virtual reality interface.

6. The method of claim 1 , wherein the virtual reality interface is a virtual reality headset.

7. The method of claim 1 , wherein the artificial intelligence model further utilizes environmental sensor information.

8. The method of claim 7 , wherein the environmental sensor information is gathered from at least one of a proximity sensor, a chemical sensor, a temperature sensor, an inertial measurement sensor, or a gyroscope.

9. A system for training an autonomous device, comprising:
a sensor configured to capture a first data corresponding to a first action taken by the autonomous device;
a virtual reality interface communicatively coupled to the sensor, the virtual reality interface configured to receive and display the first data to a human operator, wherein the first data is displayed by the virtual reality interface, and wherein the first data is labeled as representing a failed task by the human operator if the first action is not successful;
an input control configured to transmit a control signal to the autonomous device, the control signal configured to cause the autonomous device to undertake a second action, wherein the second action is associated with a second data; and
an artificial intelligence model configured to receive the first data and the second data if the first data is not labeled as representing a failed task, and further configured to analyze the first data and the second data for future control of the autonomous device,
wherein the artificial intelligence model is used to provide a control signal to the autonomous device.

10. The system of claim 9 , wherein the autonomous device is configured to perform a third action, wherein the third action is determined by analysis of the first data and the second data by the artificial intelligence model.

11. The system of claim 9 , wherein the sensor is a camera.

12. The system of claim 9 , wherein the artificial intelligence model is a deep learning model.

13. The system of claim 9 , wherein the input control is selected from a group consisting of a joystick, a keyboard, a mouse, and a game pad that is coupled to the virtual reality interface.

14. The system of claim 9 , wherein the virtual reality interface is a virtual reality headset.

15. The system of claim 9 , wherein the artificial intelligence model further utilizes environmental sensor information.

16. The system of claim 15 , wherein the environmental sensor information is gathered from at least one of a proximity sensor, a chemical sensor, a temperature sensor, an inertial measurement sensor, or a gyroscope.

17. A method for training an autonomous device, the method comprising:
capturing, using a sensor, a first data corresponding to a first action of the autonomous device;
transmitting the first data to a virtual reality interface that is viewable by a human operator wherein the first data is displayed to the human operator by using the virtual reality interface, wherein first data is labeled as representing a failed task by the human operator if the first action is not successful;
controlling the autonomous device, via an input control device by the human operator, to cause the autonomous device to undertake a second action, the second action associated with a second data;
analyzing, by an artificial intelligence model, the first data and the second data if the first data is not labeled as representing a failed task; and
generating a control signal, by the artificial intelligence model, to cause the autonomous device to undertake a third action, wherein the third action is configured to mimic an action of the human operator.

18. The method of claim 17 , wherein the artificial intelligence model is a deep learning model.

19. The method of claim 17 , wherein the input control device comprises at least one of a joystick, a keyboard, a mouse, or a game pad that is coupled to the virtual reality interface.

20. The method of claim 17 , wherein the artificial intelligence model further utilizes environmental sensor information.

Assignees

• Kindred Sys Inc

Inventors

• Gildert Suzanne
• Rose George Samuel
• Taylor Graham William
• Bergstra James

Classifications

• Y10S901/01

  Mobile robot · CPC title

• G05B2219/36442

  Automatically teaching, teach by showing · CPC title

• G05B19/42 Primary

  Recording and playback systems, i.e. in which the program is recorded from a cycle of operations, e.g. the cycle of operations being manually controlled, after which this record is played back on the same machine · CPC title

• G05B2219/36184

  Record actions of human expert, teach by showing · CPC title