Method for controlling robot and robot device

Patent metadata

Field	Value
Publication number	US-11325255-B2
Application number	US-201916668647-A
Country	US
Kind code	B2
Filing date	Oct 30, 2019
Priority date	Apr 21, 2017
Publication date	May 10, 2022
Grant date	May 10, 2022

Abstract

A method for controlling a robot includes: establishing a reference coordinate system; capturing a user's gaze direction of an indicated target; acquiring a sight line angle of the robot; acquiring a position of the robot; acquiring a linear distance between the robot and the user; calculating in real time a gaze plane in a user's gaze direction relative to the reference coordinate system based on the sight line angle of the robot, the position of the robot and the liner distance between the robot and the user; and smoothly scanning the gaze plane by the robot to search for the indicated target in the user's gaze direction.

First claim

What is claimed is:

1. A method for controlling a robot, comprising:
establishing a reference coordinate system;
capturing a user's gaze direction of an indicated target;
acquiring a sight line angle of the robot;
acquiring a position of the robot;
acquiring a linear distance between the robot and the user;
calculating in real time a gaze plane in a user's gaze direction relative to the reference coordinate system based on the sight line angle of the robot, the position of the robot and the liner distance between the robot and the user; and
smoothly scanning the gaze plane by the robot to search for the indicated target in the user's gaze direction.

2. The method according to claim 1 , wherein the robot comprises a networking module configured to network with a robot under the same coordinate system, wherein the network module is configured to share data of the gaze plane.

3. The method according to claim 2 , further comprising:
acquiring a real-time position and a real-time sight line angle of the robot by the robot during movement of the robot;
determining an intersection line between the real-time sight line angle of the robot and the gaze plane, and a focal distance; and
image-scanning a proximity region of the intersection line based on the focal distance until the indicated target gazed by the user in the gaze plane is identified.

4. The method according to claim 3 , further comprising:
extracting target characteristic information of the indicated target; and
storing the target characteristic information to a target directory.

5. The method according to claim 4 , further comprising:
acquiring voice data and video data of the user;
identifying key information of the voice data and the video data of the user;
matching the target characteristic information with the key information to obtain associated key information; and
storing the associated key information to corresponding target characteristic information in the target directory to update the target characteristic information.

6. The method according to claim 5 , further comprising:
after the robot judges that the user is chatting, collecting voice data and video data of the user's chatting;
identifying subject information of the voice data and the video data of the user's chatting;
matching the updated target characteristic information with the subject information; and
carrying out voice and video communications with the user based on a matching result.

7. A robot device, comprising:
at least one processor; and
a memory communicably connected to the at least one processor; wherein
the memory stores an instruction program executable by the at least one processor, wherein, the instruction program, when being executed by the at least one processor, cause the at least one processor to perform the steps of: establishing a reference coordinate system;
capturing a user's gaze direction of an indicated target;
acquiring a sight line angle of the robot;
acquiring a position of the robot;
acquiring a linear distance between the robot and the user;
calculating in real time a gaze plane in a user's gaze direction relative to the reference coordinate system based on the sight line angle of the robot, the position of the robot and the liner distance between the robot and the user; and
smoothly scanning the gaze plane to search for the indicated target in the user's gaze direction.

8. The robot device according to claim 7 , wherein the robot comprises a networking module configured to network with a robot under the same coordinate system, wherein the network module is configured to share data of the gaze plane.

9. The robot device according to claim 8 , wherein the instruction program, when being executed by the at least one processor, further cause the at least one processor to perform the steps of:
acquiring a real-time position and a real-time sight line angle of the robot by the robot during movement of the robot;
determining an intersection line between the real-time sight line angle of the robot and the gaze plane, and a focal distance; and
image-scanning a proximity region of the intersection line based on the focal distance until the indicated target gazed by the user in the gaze plane is identified.

10. The robot device according to claim 9 , wherein the instruction program, when being executed by the at least one processor, further cause the at least one processor to perform the steps of:
extracting target characteristic information of the indicated target; and
storing the target characteristic information to a target directory.

11. The robot device according to claim 10 , wherein the instruction program, when being executed by the at least one processor, further cause the at least one processor to perform the steps of:
acquiring voice data and video data of the user;
identifying key information of the voice data and the video data of the user;
matching the target characteristic information with the key information to obtain associated key information; and
storing the associated key information to corresponding target characteristic information in the target directory to update the target characteristic information.

12. The robot device according to claim 11 , wherein the instruction program, when being executed by the at least one processor, further cause the at least one processor to perform the steps of:
after the robot judges that the user is chatting, collecting voice data and video data of the user's chatting;
identifying subject information of the voice data and the video data of the user's chatting;
matching the updated target characteristic information with the subject information; and
carrying out voice and video communications with the user based on a matching result.

13. A computer program product comprising a software code portion, wherein the software code portion is configured to, when being run in a memory of a computer, perform steps of:
establishing a reference coordinate system;
capturing a user's gaze direction of an indicated target;
acquiring a sight line angle of the robot;
acquiring a position of the robot;
acquiring a linear distance between the robot and the user;
calculating in real time a gaze plane in a user's gaze direction relative to the reference coordinate system based on the sight line angle of the robot, the position of the robot and the liner distance between the robot and the user; and
smoothly scanning the gaze plane to search for the indicated target in the user's gaze direction.

14. The computer program product according to claim 13 , wherein the robot comprises a networking module configured to network with a robot under the same coordinate system, wherein the network module is configured to share data of the gaze plane.

15. The computer program product according to claim 14 , wherein the software code portion is further configured to, when being run in a memory of a computer, perform steps of:
acquiring a real-time position and a real-time sight line angle of the robot by the robot during movement of the robot;
determining an intersection line between the real-time sight line angle of the robot and the gaze plane, and a focal distance; and
image-scanning a proximity region of the intersection line based on the focal distance until the indicated target gazed by the user in the gaze plane is identified.

16. The computer program product according to claim 15 , wherein the software code portion is further configured to, when being run in a memory of a computer, perform steps of:
extracting target characteristic informati

Assignees

• Cloudminds Shenzhen Robotics Systems Co Ltd
• Cloudminds Robotics Co Ltd

Inventors

• Luo Lei

Classifications

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• B25J9/1656

  characterised by programming, planning systems for manipulators · CPC title

• G05B2219/40411

  Robot assists human in non-industrial environment like home or office · CPC title

• G06V40/18

  Eye characteristics, e.g. of the iris · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title