Digital twin modeling method and system for assembling a robotic teleoperation environment

Patent metadata

Field	Value
Publication number	US-12243159-B2
Application number	US-202217893368-A
Country	US
Kind code	B2
Filing date	Aug 23, 2022
Priority date	Jan 20, 2022
Publication date	Mar 4, 2025
Grant date	Mar 4, 2025

Abstract

A digital twin modeling method to assemble a robotic teleoperation environment, including: capturing images of the teleoperation environment; identifying a part being assembled; querying the assembly assembling order to obtain a list of assembled parts according to the part being assembled; generating a three-dimensional model of the current assembly from the list and calculating position pose information of the current assembly in an image acquisition device coordinate system; loading a three-dimensional model of the robot, determining a coordinate transformation relationship between a robot coordinate system and an image acquisition device coordinate system; determining position pose information of the robot in an image acquisition device coordinate system from the coordinate transformation relationship; determining a relative positional relationship between the current assembly and the robot from position pose information of the current assembly and the robot in an image acquisition device coordinate system; establishing a digital twin model of the teleoperation environment.

First claim

The invention claimed is:

1. A digital twin modeling method of assembling a robotic teleoperation environment, comprising the steps of:
capturing, by an image acquisition device, an image of a teleoperation environment;
identifying a part currently being assembled in the image;
querying the assembly assembling order to obtain a list of assembled parts according to the part currently being assembled;
generating a three-dimensional model of the current assembly from the list of assembled parts and calculating position pose information of the current assembly in an image acquisition device coordinate system;
loading a three-dimensional model of the robot, determining a coordinate transformation relationship between a robot coordinate system and an image acquisition device coordinate system;
position calibrating the robot in the image according to the coordinate transformation relationship, and determining position pose information of the robot in an image acquisition device coordinate frame;
determining a relative positional relationship between the current assembly and the robot from the position pose information of the current assembly in the image acquisition device coordinate system and the position pose information of the robot in the image acquisition device coordinate system;
establishing a digital twin model of the teleoperation environment based on a three-dimensional model of the current assembly, a three-dimensional model of the robot, and a relative positional relationship between the current assembly and the robot.

2. The digital twin modeling method of assembling a robotic teleoperation environment according to claim 1 , wherein the images captured by the image acquisition device include depth images, the step of the identifying parts currently being assembled in the images is specified by:
identifying regions of different parts of the assembly in the depth image using different color labels to generate an instance segmented image;
counting color labels of the example segmented images to identify the parts that are currently being assembled.

3. The digital twin modeling method of assembling a robotic teleoperational environment according to claim 2 , wherein the step of the generating a three-dimensional model of the current assembly from the list of assembled parts is specified by:
loading a three-dimensional model of each part in the list of assembled parts;
determining positions of the parts on the assembly in the list according to predefined constraint relationships between the parts of the assembly;
generating the three-dimensional model of the current assembly by adjusting the position of the three-dimensional model of each part according to the position of each part in the assembly in the list.

4. The digital twin modeling method of assembling a robotic teleoperation environment according to claim 2 , wherein the step of the calculating position pose information of the current assembly in an image acquisition device coordinate system is specified by:
pre-processing the depth image, removing background, preserving the depth image of the current assembly;
converting the depth image of the current assembly to an assembly point cloud with the intrinsic and imaging model of the image acquisition device;
inputting the assembly point cloud to a point cloud feature extraction network to extract point cloud features of the assembly;
inputting the point cloud features of the assembly to a pre-trained pose estimation network, outputting position pose information of the assembly in the image acquisition device coordinate system.

5. The digital twin modeling method of assembling a robotic teleoperation environment according to claim 4 , wherein the pre-training of the pose estimation network is:
determining initial information; extracting and recording label position pose information of the three-dimensional model point cloud of the assembly at the initial perspective, the label position pose information comprising a rotation matrix R i and an offset matrix T i , for each point in the three-dimensional model point cloud, i being an index for each point in the three-dimensional model point cloud;
point cloud conversion; extracting an assembly depth image of the 3D model of the assembly at another view angle different from the initial view angle and converting the assembly depth image into an assembly point cloud using the intrinsic and imaging model of the image acquisition device;
pose prediction; inputting the assembly point cloud to a point cloud feature extraction network, outputting point cloud point-wise features, inputting the point cloud point-wise features to a pose estimation network, predicting pose prediction information for each point comprising a predicted rotation matrix R′ i and a predicted offset matrix T′ i ;
calculating a Euclidean distance of the pose prediction information of each point from the label position pose information, generating confidence based on the Euclidean distance; performing a step of the image update if the confidence is less than a set threshold and outputting assembly optimal pose prediction information for the current view angle if the confidence is greater than the set threshold or the number of trains reaches a set value; determining whether training is complete, returning to the step of the point cloud conversion if not complete, continuing training by replacing the assembly depth image at the next view angle, and performing the step of the image update if complete;
an image update; performing displacement and rotation transformations on the assembly point cloud, using the predicted rotation matrix R′ i and the predicted offset matrix T′ i as inputs, updating the three-dimensional coordinates of the assembly point cloud, and inputting the updated assembly point cloud to the point cloud feature extraction network to continue training.

6. The digital twin modeling method of assembling a robotic teleoperation environment according to claim 2 , wherein the image acquisition device is an RGB-D camera, the captured images further includes RGB images; the step of the determining a coordinate transformation relationship between the robot coordinate system and the image acquisition device coordinate system is specified by:
disposing a positioning marker at a joint of the robot;
controlling the robot to make point-position intermittent motions, at each point of intermittent motion, reading coordinates P i (x wi , y wi , z wi ) of the positioning marker in the robot coordinate system by the robot controller while identifying pixel coordinates Z i (u i , v i ) of the feature point of the positioning marker in the RGB image;
deriving three-dimensional coordinates P′ i (x ci , y ci , z ci ) of the positioning marker's feature point in the camera coordinate system using the positional relationship between the depth camera and the color camera in the RGB-D camera based on the pixel coordinates Z i (u i , v i ) of the positioning marker's feature point in the RGB image and the imaging model of the RGB-D camera;
obtaining the coordinates P i (x wi , y wi , z wi ) of the positioning marker in the robot coordinate system and the three-dimensional coordinates P′ i (x ci , y ci , z ci ) of the feature points of the positioning marker in the camera coordinate system of the four or more intermittent motion points, and calculating the transformation matrices of the robot coordinate system and the camera coordinate system.

7. The digital twin modeling method of assembling a robotic teleoperation environment according to claim 6 , wherein in the step of controlling the robot to make the point-position intermittent motion, a trajectory of the set point-position intermittent motion is a plurality of squares.

8

Assignees

• Univ Qingdao Technology
• Univ Shandong
• Univ Hebei Technology
• Ebei Univ Of Technology

Inventors

• Chen Chengjun
• Zhao Zhengxu
• Hu Tianliang
• Zhang Jianhua
• Guo Yang
• Li Dongnian
• Zhang Qinghai
• Guan Yuanlin

Classifications

• G06T2219/2004

  Aligning objects, relative positioning of parts · CPC title

• G06T2207/30204

  Marker · CPC title

• G06T2207/30164

  Workpiece; Machine component · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title