Vision-based operation for robot

Patent metadata

Field	Value
Publication number	US-11926065-B2
Application number	US-202016941988-A
Country	US
Kind code	B2
Filing date	Jul 29, 2020
Priority date	Feb 1, 2018
Publication date	Mar 12, 2024
Grant date	Mar 12, 2024

Abstract

Embodiments of the present disclosure relates to an electronic device, a vision-based operation method and system for a robot. The electronic device includes at least one processing unit; and a memory coupled to the at least one processing unit and storing computer program instructions therein, the instructions, when executed by the at least one processing unit, causing the electronic device to perform acts including: obtaining an image containing a tool of a robot and an object to be operated by the tool, the image being captured by a camera with a parameter, obtaining a command for operating the tool, the command generated based on the image; and controlling the robot based on the command and the parameter. Embodiments of the present disclosure can greatly improve the accuracy, efficiency and safety of an operation of a robot.

First claim

What is claimed is:

1. An electronic device, comprising:
at least one processing unit; and
a memory coupled to the at least one processing unit and storing computer program instructions therein, the instruction, when executed by the at least one processing unit, causing the electronic device to:
obtain an image containing a tool of a robot and an object to be operated by the tool, the image being captured by a camera with a parameter;
obtain signals generated by sensors arranged on the robot at one or more locations including joints of the robot;
generate a further image showing a posture of the robot based on the signals;
obtain a command for operating the tool, the command generated based on the image and the further image;
determine a field of view of the image corresponding to the parameter;
control the robot based on the command and the parameter; and
wherein the parameter is related to a size of the field of view of the camera,
wherein controlling the robot comprises:
in response to the size decreasing, controlling the tool to move with a decreased speed; and
in response to the size increasing, controlling the tool to move with an increased speed.

2. The electronic device of claim 1 , wherein the instruction, when executed by the at least one processing unit, causes the image to be displayed to a user.

3. The electronic device of claim 1 , wherein the parameter is related to a distance between the camera and the tool, and wherein controlling the robot comprises:
in response to the distance decreasing, controlling the tool to move with a decreased speed; and
in response to the distance increasing, controlling the tool to move with an increased speed.

4. The electronic device of claim 1 , wherein the instructions, when executed by the at least one processing unit, cause the electronic device to:
cause the further image to be displayed to a user.

5. The electronic device of claim 1 , wherein obtaining the image comprises obtaining a video containing the tool and the object.

6. The electronic device of claim 1 , wherein the image has a focus on a predefined reference point of the tool.

7. A vision-based operation method for a robot, the method comprising:
obtaining a command for operating a tool, the command generated based on an image;
determining a field of view of the image corresponding to a parameter;
obtaining signals generated by sensors arranged on the robot at one or more locations including joints of the robot;
generating a further image showing a posture of the robot based on the signals; and
controlling the robot based on the command and the parameter and the further image;
wherein the parameter is related to a size of the field of view of a camera, and wherein controlling the robot comprises:
in response to the size decreasing, controlling the tool to move with a decreased speed; and
in response to the size increasing, controlling the tool to move with an increased speed.

8. The method of claim 7 , further comprising:
causing the image to be displayed to a user.

9. The method of claim 7 , wherein the parameter is related to a distance between the camera and the tool, and wherein controlling the robot comprises:
in response to the distance decreasing, controlling the tool to move with a decreased speed; and
in response to the distance increasing, controlling the tool to move with an increased speed.

10. The method of claim 7 , further comprising:
causing the further image to be displayed to a user.

11. The method of claim 7 , wherein obtaining the image comprises obtaining a video containing the tool and an object.

12. The method of claim 7 , wherein the image has a focus on a predefined reference point of the tool.

13. A vision-based operation system for a robot, comprising:
an electronic device comprising:
a camera;
at least one processing unit; and
a memory coupled to the at least one processing unit and storing computer program instructions therein, the instruction, when executed by the at least one processing unit, causing the electronic device to:
obtain an image containing a tool of a robot and an object to be operated by the tool, the image being captured by the camera;
obtain signals generated by sensors arranged on the robot including a joint of the robot;
generate a further image showing a posture of the robot based on the signals;
obtain a command for operating the tool, the command generated based on the image and the further image;
determine a field of view of the image;
control the robot based on the command and a parameter; and
wherein the parameter is related to a size of the field of view of the camera,
wherein controlling the robot comprises:
in response to the size decreasing, controlling the tool to move with a decreased speed; and
in response to the size increasing, controlling the tool to move with an increased speed;

wherein the camera is configured to capture the image containing the tool of the robot and the object with the parameter.

14. The system of claim 13 , further comprising a display configured to display the image and the further image to a user.

15. The system of claim 14 , further comprising an input device configured to receive the command and transmit the command to the electronic device.

16. A computer program product tangibly stored in a non-transient computer readable medium and including machine executable instructions which, when executed, cause a robot to obtain an image containing a tool of the robot and an object to be operated by the tool, the image being captured by a camera with a parameter;
obtain a command for operating the tool, the command generated based on the image;
obtain signals generated by sensors arranged on the robot including a joint of the robot;
generate a further image showing a posture of the robot based on the signals; and
control the robot based on the command and the parameter and the further image;
wherein the parameter is related to a size of a field of view of the camera, and wherein controlling the robot comprises:
in response to the size decreasing, controlling the tool to move with a decreased speed; and
in response to the size increasing, controlling the tool to move with an increased speed.

Assignees

• Abb Schweiz Ag

Inventors

• Pan Wengui
• Ma Bojun
• Wang Yanjun

Classifications

• B25J9/1697 Primary

  Vision controlled systems · CPC title

• B25J9/1651

  acceleration, rate control · CPC title

• B25J9/1664

  characterised by motion, path, trajectory planning · CPC title

• B25J9/1689

  Teleoperation · CPC title

• B25J19/023

  including video camera means · CPC title