Method for planning a movement path for a robotic arm

Patent metadata

Field	Value
Publication number	US-12528187-B2
Application number	US-202418627893-A
Country	US
Kind code	B2
Filing date	Apr 5, 2024
Priority date	Nov 27, 2023
Publication date	Jan 20, 2026
Grant date	Jan 20, 2026

Abstract

A method is provided for planning a movement path for a non-fixed end of a robotic arm. A mixed reality (MR) device is used to capture hand images of a user to obtain multiple target coordinate sets in an MR coordinate system. A computer device is used to convert the target coordinate sets into multiple path coordinate sets in a robot base coordinate system, and controls the non-fixed end of a robotic arm to move along a movement path as indicated by the path coordinate sets. A camera device is used to capture images of a base that the robotic arm is fixed to for conversion of the target coordinate sets into the path coordinate sets.

First claim

What is claimed is:

1 . A method for planning a movement path for a non-fixed end of a robotic arm, the robotic arm having a fixed end that is opposite to the non-fixed end and that is fixed to a base, the method comprising:
providing a robotic arm control system that is communicatively connected to a camera device which is configured to capture images of the base, wherein the robotic arm control system includes:
a mixed reality (MR) device that is to be worn on a user; and
a computer device that is communicatively connected to the robotic arm and the MR device, and that stores a first transformation matrix and a second transformation matrix,
the first transformation matrix being used to perform coordinate transformation from a robot base coordinate system related to the fixed end of the robotic arm to a navigation coordinate system related to the camera device,
the second transformation matrix being used to perform coordinate transformation from the navigation coordinate system to an MR coordinate system related to the MR device;
the method further comprising:
A) by the MR device, capturing a target-setting hand image that is related to a hand of the user, and obtaining a target coordinate set in the MR coordinate system based on the target-setting hand image, where the target coordinate set is related to the hand of the user;
B) by the MR device, determining whether an instruction for terminating path planning is received;
C) by the MR device, repeating step A) upon determining that the instruction for terminating path planning has not been received yet;
D) by the MR device, after determining that the instruction for terminating path planning is received, generating and sending an execution request to the computer device, the execution request including the target coordinate sets that are obtained respectively in multiple executions of step A) and that are arranged in a user-defined order;
E) by the computer device, upon receipt of the execution request, converting, based on the first transformation matrix and the second transformation matrix, the target coordinate sets in the MR coordinate system respectively to path coordinate sets in the robot base coordinate system, the path coordinate sets being arranged in the user-defined order and cooperatively indicating a movement path; and
F) by the computer device, controlling the non-fixed end of the robotic arm to move along the movement path by generating and sending to the robotic arm an operating instruction that includes the path coordinate sets.

2 . The method as claimed in claim 1 , wherein step A) includes:
A-1) by the MR device, capturing a user-hand image, which is an image of the hand of the user, and acquiring a plurality of hand node coordinate sets in the MR coordinate system based on the user-hand image, the hand node coordinate sets being related to a plurality of hand nodes of the hand, respectively; A-2) by the MR device, determining whether a first portion of the hand node coordinate sets indicates a first gesture; and A-3) by the MR device, upon determining that the first portion of the hand node coordinate sets indicates the first gesture, making the user-hand image serve as the target-setting hand image, and acquiring the target coordinate set based on the first portion of the hand node coordinate sets.

3 . The method as claimed in claim 2 , wherein:
in sub-step A-1), the hand node coordinate sets include a first fingertip coordinate set that corresponds to a first fingertip of the hand, and a second fingertip coordinate set that corresponds to a second fingertip of the hand; in sub-step A-2), the first portion of the hand node coordinate sets includes the first fingertip coordinate set and the second fingertip coordinate set, and the MR device determines whether the first portion of the hand node coordinate sets indicates the first gesture by determining whether a distance between the first fingertip coordinate set and the second fingertip coordinate set is smaller than a first predetermined distance; and in sub-step A-3), the target coordinate set is one of the first fingertip coordinate set and the second fingertip coordinate set.

4 . The method as claimed in claim 3 , wherein:
sub-step A-1) further includes, by the MR device, acquiring a hand rotational angle data piece based on the target-setting hand image, the hand rotational angle data piece being related to the hand of the user and corresponding to the hand node coordinate sets; sub-step A-3) further includes, by the MR device, upon determining that the first portion of the hand node coordinate sets indicates the first gesture, storing a rotational angle data piece that corresponds to the target coordinate set, the rotational angle data piece including a pitch angle, a yaw angle and a roll angle with respect to the MR coordinate system; and said method further comprises:
G) by the MR device, determining whether a second portion of the hand node coordinate sets indicates a second gesture that is different from the first gesture upon determining in sub-step A-2) that the first portion of the hand node coordinate sets does not indicate the first gesture, the second portion of the hand node coordinate sets including a reference node coordinate set;
H) by the MR device, upon determining in step G) that the second portion of the hand node coordinate sets indicates the second gesture, determining whether at least one target coordinate set has been obtained;
I) by the MR device, determining whether the at least one target coordinate set includes a to-be-adjusted coordinate set upon determining in step H) that at least one target coordinate set has been obtained, where a distance between the to-be-adjusted coordinate set and the reference node coordinate set is smaller than a second predetermined distance and is a minimum of all distance(s) each between the reference node coordinate set and a corresponding one of the at least one target coordinate set;
J) by the MR device, upon determining in step I) that the at least one target coordinate set includes the to-be-adjusted coordinate set, determining whether a first angle adjustment data piece is found in the MR device, the first angle adjustment data piece being used to update the rotational angle data piece that corresponds to the to-be-adjusted coordinate set;
K) by the MR device, upon determining in step J) that the first angle adjustment data piece is not found in the MR device, making the hand rotational angle data piece serve as the first angle adjustment data piece, and performing step B);
L) by the MR device, upon determining in step J) that the first angle adjustment data piece is found in the MR device, making the hand rotational angle data piece serve as the second angle adjustment data piece;
M) by the MR device, updating the to-be-adjusted coordinate set based on the reference node coordinate set, and updating the rotational angle data piece that corresponds to the to-be-adjusted coordinate set based on the first angle adjustment data piece and the second angle adjustment data piece, followed by deleting the second angle adjustment data piece, and performing step B); and
N) by the MR device, deleting the first angle adjustment data piece and the second angle adjustment data piece upon determining in step G) that the second portion of the hand node coordinate sets does not indicate the second gesture.

5 . The method as claimed in claim 4 , wherein the hand node coordinate sets further include a hand joint coordinate set indicating a joint of the hand, and a wrist joint coordinate set indicating a wrist joint of the hand, the second portion of the hand node coordinate sets includes the hand joint coordinate set, the wrist joint coordinate set and the second fingertip coordinate set; and
wherein ste

Assignees

• National Yang Ming Chiao Tung Univ

Inventors

• Song Kai-Tai
• Chiu Shih-Wei

Classifications

• B25J9/1697 Primary

  Vision controlled systems · CPC title

• B25J13/08

  by means of sensing devices, e.g. viewing or touching devices · CPC title

• B25J9/1664 Primary

  characterised by motion, path, trajectory planning · CPC title