Dual hand detection in teaching from demonstration

Patent metadata

Field	Value
Publication number	US-11712797-B2
Application number	US-202017018674-A
Country	US
Kind code	B2
Filing date	Sep 11, 2020
Priority date	Sep 11, 2020
Publication date	Aug 1, 2023
Grant date	Aug 1, 2023

Abstract

A method for dual hand detection in robot teaching from human demonstration. A camera image of the demonstrator's hands and workpieces is provided to a first neural network which determines the identity of the left and right hand of the human demonstrator from the image, and also provides cropped sub-images of the identified hands. The first neural network is trained using images in which the left and right hands are pre-identified. The cropped sub-images are then provided to a second neural network which detects the pose of both the left and right hand from the images, where the sub-image for the left hand is horizontally flipped before and after the hand pose detection if second neural network is trained with right hand images. The hand pose data is converted to robot gripper pose data and used for teaching a robot to perform an operation through human demonstration.

First claim

What is claimed is:

1. A method for dual hand detection in images, said method comprising:
providing an image including left and right hands of a human;
analyzing the image, using a first neural network running on a computer having a processor and memory, to determine an identity and a location in the image of the left hand and the right hand;
creating a left hand sub-image and a right hand sub-image, where each of the sub-images is cropped from the image;
providing the sub-images to a second neural network running on the computer, including horizontally flipping either the left hand sub-image or the right hand sub-image;
analyzing the sub-images by the second neural network to determine three-dimensional (3D) coordinates of a plurality of key points on the left and right hands; and
using the 3D coordinates of the key points by a robot teaching program to define gripper poses, including horizontally flipping the coordinates of the key points on either the left hand or the right hand.

2. The method according to claim 1 wherein the image is provided by a two-dimensional (2D) digital camera.

3. The method according to claim 1 wherein the first neural network is trained to distinguish the left hand from the right hand in a training process where a plurality of training images are provided to the first neural network in which left and right hands are pre-identified.

4. The method according to claim 3 wherein the first neural network analyzes the training images to identify distinguishing characteristics of left hands and right hands, including curvature and relative locations of digits.

5. The method according to claim 1 wherein each of the sub-images is cropped to include the left or right hand within a predefined margin.

6. The method according to claim 1 wherein horizontally flipping either the left hand sub-image or the right hand sub-image includes horizontally flipping the left hand sub-image when the second neural network is trained using training images of right hands, and horizontally flipping the right hand sub-image when the second neural network is trained using training images of left hands.

7. The method according to claim 1 wherein the plurality of key points on the left and right hands include thumb tips, thumb knuckles, finger tips and finger knuckles.

8. The method according to claim 1 wherein horizontally flipping the coordinates of the key points on either the left hand or the right hand includes horizontally flipping the coordinates of the key points on the hand which had its sub-image flipped before analysis by the second neural network.

9. The method according to claim 8 wherein horizontally flipping the coordinates of the key points includes horizontally flipping the coordinates across a vertical plane to restore the coordinates to their position in the image.

10. The method according to claim 1 wherein the image also includes one or more workpieces, and the gripper poses and workpiece positions and poses are used by the robot teaching program to create workpiece pick-up and placement instructions for a robot.

11. The method according to claim 10 wherein the instructions are provided to a robot controller from the computer, and the robot controller provides control commands to the robot to perform workpiece operations.

12. A method for programming a robot to perform an operation by human demonstration, said method comprising:
demonstrating the operation on workpieces by a human using both hands;
analyzing camera images of the hands demonstrating the operation on the workpieces, by a computer, to create demonstration data including gripper poses computed from three-dimensional (3D) coordinates of key points of the hands, where the 3D coordinates of the key points are determined from the images by a first neural network used to identify left and right hands in the images and a second neural network used to compute the 3D coordinates in sub-images of the identified left and right hands;
generating robot motion commands, based on the demonstration data, to cause the robot to perform the operation on the workpieces; and
performing the operation on the workpiece by the robot.

13. The method according to claim 12 wherein the demonstration data includes, at a grasping step of the operation, position and orientation of a hand coordinate frame, a gripper coordinate frame corresponding to the hand coordinate frame, and a workpiece coordinate frame.

14. The method according to claim 12 wherein the first neural network is trained to distinguish the left hand from the right hand in a training process where a plurality of training images are provided to the first neural network in which left and right hands are pre-identified.

15. The method according to claim 12 wherein either the left hand sub-images or the right hand sub-images are horizontally flipped before being provided to the second neural network, and the 3D coordinates of the key points of the left hand or the right hand are horizontally flipped after being computed by the second neural network.

16. The method according to claim 15 wherein the left hand sub-images and the 3D coordinates of the key points of the left hand are horizontally flipped when the second neural network is trained using training images of right hands.

17. A system for dual hand detection in images used to program a robot to perform an operation by human demonstration, said system comprising:
a camera;
a computer having a processor and memory and in communication with the camera, said computer being configured to perform steps including;
analyzing an image including left and right hands of a human, using a first neural network, to determine an identity and a location in the image of the left hand and the right hand;
creating a left hand sub-image and a right hand sub-image, where each of the sub-images is cropped from the image;
providing the sub-images to a second neural network running on the computer, including horizontally flipping either the left hand sub-image or the right hand sub-image;
analyzing the sub-images by the second neural network to determine three-dimensional (3D) coordinates of a plurality of key points on the left and right hands; and
using the 3D coordinates of the key points to define gripper poses used to program the robot, including horizontally flipping the coordinates of the key points on either the left hand or the right hand.

18. The system according to claim 17 wherein the first neural network is trained to distinguish the left hand from the right hand in a training process where a plurality of training images are provided to the first neural network in which left and right hands are pre-identified, and where the first neural network analyzes the training images to identify distinguishing characteristics of left hands and right hands, including curvature and relative locations of digits.

19. The system according to claim 17 wherein horizontally flipping either the left hand sub-image or the right hand sub-image includes horizontally flipping the left hand sub-image when the second neural network is trained using training images of right hands, and horizontally flipping the right hand sub-image when the second neural network is trained using training images of left hands.

20. The system according to claim 19 wherein horizontally flipping the coordinates of the key points on either the left hand or the right hand includes horizontally flipping the coordinates of the key points on the hand which had its sub-image flipped before an

Assignees

• Fanuc Corp

Inventors

• Wang Kaimeng
• Kato Tetsuaki

Classifications

• B25J9/0081 Primary

  with leader teach-in means · CPC title

• B25J9/161

  Hardware, e.g. neural networks, fuzzy logic, interfaces, processor · CPC title

• B25J9/1612 Primary

  characterised by the hand, wrist, grip control · CPC title

• B25J13/08

  by means of sensing devices, e.g. viewing or touching devices · CPC title

• B25J19/023

  including video camera means · CPC title