System and method for robot teaching based on RGB-D images and teach pendant

Patent metadata

Field	Value
Publication number	US-11440179-B2
Application number	US-202016804888-A
Country	US
Kind code	B2
Filing date	Feb 28, 2020
Priority date	Jul 23, 2019
Publication date	Sep 13, 2022
Grant date	Sep 13, 2022

Abstract

A system for robot teaching based on RGB-D images and a teach pendant, including an RGB-D camera, a host computer, a posture teach pendant, and an AR teaching system which includes an AR registration card, an AR module, a virtual robot model, a path planning unit and a posture teaching unit. The RGB-D camera collects RGB images and depth images of a physical working environment in real time. In the path planning unit, path points of a robot end effector are selected, and a 3D coordinates of the path points in the basic coordinate system of the virtual robot model are calculated; the posture teaching unit records the received posture data as the postures of a path point where the virtual robot model is located, so that the virtual robot model is driven to move according to the postures and positions of the path points, thereby completing the robot teaching.

First claim

What is claimed is:

1. A system for robot teaching based on RGB-D images and a teach pendant, comprising:
a RGB-D camera,
a host computer,
a posture teach pendant, and
an AR teaching system;
wherein, the RGB-D camera and the posture teach pendant are communicated with the host computer; the RGB-D camera is set in a physical working environment, and the AR teaching system comprises an AR registration card located in the physical working environment, an AR module running in the host computer, a virtual robot model comprising a robot end effector, a path planning unit and a posture teaching unit;
RGB images and depth images of the physical working environment are collected by the RGB-D camera in real time and are sent to the host computer; the AR module sets a position of a virtual camera in a virtual scene, and overlays the virtual robot model on the RGB images to complete AR registration; the path planning unit displays the RGB images and depth images, and a teaching programmer interactively selects path points of the robot end effector on the RGB images, thereby calculating a 3D coordinate of each of the path points of the robot end effector in a basic coordinate system of the virtual robot model based on a transformation between the depth images and the RGB images; the posture teach pendant is operated by the teaching programmer to generate posture data; the posture teaching unit receives the posture data of the posture teach pendant in real time when the host computer sequentially reads the 3D coordinates of the path points of the robot end effector in the basic coordinate system of the virtual robot model, and then the virtual robot model moves based on the 3D coordinates; during the movement, the received posture data is recorded as the posture data corresponding to a 3D coordinate of a path point where the virtual robot model is located, so that the virtual robot model moves in accordance with teaching postures and positions to complete the robot teaching.

2. The system of claim 1 , wherein, the AR teaching system further comprises a virtual-real collision detection module running in the host computer; when the virtual robot model moves in accordance with the teaching postures and positions, the virtual-real collision detection module detects whether the virtual robot model interferes with the physical working environment; if yes, a prompt is issued, so that the teaching programmer controls the posture teach pendant to adjust the teaching postures and positions in time until the interference disappears;
after the robot teaching is completed, the path points are edited to form a path trajectory; based on the edited path trajectory, the host computer generates a program code based on an instruction format of a programming system of a physical robot, and transmits the program code to a controller of the physical robot, so as to control the physical robot to work in the physical working environment.

3. The system of claim 1 , wherein, the posture teach pendant comprises a signal processing unit, an inertial measurement unit connected to the signal processing unit, an input switch button unit, a wireless communication unit, and an interface display unit; an inertial sensor built in the inertial measurement unit is used to measure the posture data of the posture teach pendant in a Cartesian coordinate system; the teaching programmer sets parameter information of the robot end effector through the input switch button unit, and the input switch button unit transmits the parameter information set by a button to the signal processing unit; the parameter information and posture data are processed by the signal processing unit and sent by the wireless communication unit to the host computer, and the host computer drives the virtual robot model after receiving the processed parameter information and posture data; and the interface display unit displays working states and working data of the posture teach pendant in real time.

4. The system of claim 3 , wherein, the posture teaching unit receives the posture data and parameter information of the posture teach pendant in real time when the host computer sequentially reads the 3D coordinates of the path points in the basic coordinate system of the virtual robot model, and then the virtual robot model moves based on the 3D coordinates; during the movement, the received posture data and the parameter information are recorded as the posture data and the parameter information corresponding to 3D coordinates of the path point where the virtual robot model is located, so that the virtual robot model moves in accordance with the teaching postures and positions to complete the robot teaching.

5. The system of claim 1 , wherein, the virtual robot model is established in the host computer; specifically, in the host computer, 3D models of the physical robot and the robot end effector which are the same as the physical robot are drawn to scale; and a forward kinematics model and an inverse kinematics model are established based on structures and parameters of the physical robot and the robot end effector, thereby establishing the virtual robot model.

6. The system of claim 5 , wherein, after creating the virtual robot model and completing the AR registration, the host computer further performs the following steps:
(1) establishing coordinate systems and calibrating mapping relationships;
(i) setting a coordinate system of the AR registration card as a world coordinate system of the virtual scene and a physical scene, and setting the coordinate system of the AR registration card as the basic coordinate system of the virtual robot model; making a basic coordinate system of the physical robot coincide with the basic coordinate system of the virtual robot model;
(ii) in the path planning unit: establishing an RGB image pixel coordinate system, a depth image pixel coordinate system, an RGB camera coordinate system, and a depth camera coordinate system, and establishing mapping relationships among the RGB image pixel coordinate system, the depth image pixel coordinate system, the RGB camera coordinate system, and the depth camera coordinate systems; and
(iii) in the posture teaching unit: obtaining a homogeneous transformation matrix M V N between the basic coordinate system of the virtual robot model and the Cartesian coordinate system through calibrating or setting; where the basic coordinate system of the posture teach pendant is the Cartesian coordinate system; Q V =M V N *Q, where Q is the homogeneous coordinate matrix of the posture teach pendant in the Cartesian coordinate system, and Q V is the homogeneous coordinate matrix of the posture teach pendant in the basic coordinate system of the virtual robot model, converting the posture data of the posture teach pendant in the Cartesian coordinate system to the basic coordinate system of the virtual robot model.

7. The system of claim 1 , wherein, the step of calculating the 3D coordinate of each of the path points of the robot end effector in the basic coordinate system of the virtual robot model based on the transformation between the depth images and the RGB images comprises:
after the teaching programmer selects the path points on the RGB images, calibrating a RGB image pixel coordinate (u, v) of each of the path points; calculating a transformation matrix H of the RGB image pixel coordinate system relative to the depth image pixel coordinate system through the calibration or factory parameters of the RGB-D camera, and calculating a depth image pixel coordinate (u, v d ) of each of the path points corresponding to the RGB image pixel coordinate (u,v) based on the transformation matrix H, and reading out a corresponding depth value z; calibrating an internal parameter matrix M of the depth camera, and

Assignees

• Univ Qingdao Technology

Inventors

• Chen Chengjun
• Pan Yong
• Li Dongnian
• Hong Jun

Classifications

• B25J9/1697

  Vision controlled systems · CPC title

• B25J19/023

  including video camera means · CPC title

• G05B2219/36492

  Record position and orientation, posture of probe, tool · CPC title

• B25J9/0081 Primary

  with leader teach-in means · CPC title

• B25J9/1671

  characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems · CPC title