Moving target following method, robot and computer-readable storage medium

Patent metadata

Field	Value
Publication number	US-12093054-B2
Application number	US-202117239603-A
Country	US
Kind code	B2
Filing date	Apr 25, 2021
Priority date	Apr 25, 2021
Publication date	Sep 17, 2024
Grant date	Sep 17, 2024

Abstract

A moving target following method, which is executed by one or more processors of a robot that includes a camera and a sensor electrically coupled to the one or more processors, includes: performing a body detection to a body of a target based on images acquired by the camera to obtain a body detection result; performing a leg detection to legs of the target based on data acquired by die sensor to obtain a leg detection result; and fusing the body detection result and the leg detection result to obtain a fusion result, and controlling the robot to follow the target based on the fusion result.

First claim

What is claimed is:

1. A computer-implemented following method executed by one or more processors to control a robot that comprises a camera and a sensor electrically coupled to the one or more processors, the method comprising:
performing a body detection to a body of a target based on images acquired by the camera to obtain a body detection result;
performing a leg detection to legs of the target based on data acquired by the sensor to obtain a leg detection result;
fusing the body detection result and the leg detection result to obtain a fusion result, and controlling the robot to follow the target based on the fusion result;
determining whether disappearance of the target is detected;
in response to detection of the disappearance of the target, determining a position of the target relative to the robot before disappearance of the target as a desired position, according to a last frame of data obtained by the camera and/or the sensor before the disappearance of the target;
determining whether there is an obstacle between the desired position and the robot;
in response to no obstacle being detected, determining a direction in which the target disappears according to the desired position, and controlling the robot to rotate according to the direction in which the target disappears, wherein a rotation direction of the robot is consistent with the direction in which the target disappears; and
controlling the robot to stop moving, in response to no target being detected after the rotation of the robot.

2. The method of claim 1 , wherein performing the body detection to the body of the target based on images acquired by the camera to obtain the body detection result comprises:
identifying candidate objects in the images using a pre-trained neural network model, wherein the candidate objects have preset body skeleton features, the preset body skeleton features comprise features of a head, shoulders, arms, a body and legs, and the images are color images with depth information;
determining the target from the candidate objects through facial recognition according to pre-stored facial features of the target;
extracting identification features on the body of the target from the images, wherein the identification features on the body of the target comprise: color, texture and text on clothes of the target, decorations on the target, or appearance characteristics of the target; and
determining, according to the identification features and external parameters of the camera relative to robot coordinates, position information of the body of the target in a robot coordinate system as the body detection result, wherein the external parameters comprise a relative pose of the camera.

3. The method of claim 1 , wherein the sensor comprises a two-dimensional lidar sensor, and performing the leg detection to legs of the target based on data acquired by the sensor to obtain the leg detection result comprises:
acquiring two-dimensional lidar data from the two-dimensional lidar sensor, and dividing a two-dimensional point cloud in the two-dimensional lidar data to obtain a plurality of continuous line segments;
determining line segments of the plurality of continuous line segments, whose shape similarity with shape of the legs of the target is greater than a preset first similarity, to obtain a plurality of candidate legs;
identifying at least two legs of the plurality of candidate legs whose shape similarity is greater than a preset second similarity and a distance between the at least two legs is greater than a preset distance as belonging to a same object;
determining, according to position information of the at least two legs, position information of the legs of the target as the leg detection result, wherein position information of one of the at least two legs is determined as the leg detection result, or position information of center positions of the at least two legs is determined as the leg detection result.

4. The method of claim 1 , further comprising, before fusing the body detection result and the leg detection result to obtain the fusion result, and controlling the robot to follow the target based on the fusion result, checking the body detection result and the leg detection result, wherein fusing the body detection result and the leg detection result to obtain the fusion result comprises: fusing the body detection result and the leg detection result after they have been checked.

5. The method of claim 4 , wherein checking the body detection result comprises:
determining, according to the body detection result, whether a position of the target is within a preset first position range, and whether a linear velocity and an acceleration of the target are respectively less than a preset linear velocity threshold and a preset acceleration threshold, wherein the preset first position range is a best field of view and a best range of operation of the camera;
determining that the target has been detected, in response to the position of the target being within the preset first position range, and the linear velocity and the acceleration of the target being respectively less than the preset linear velocity threshold and the preset acceleration threshold; and
determining that the target is not detected, in response to the position of the target not being within the preset first position range, or the linear velocity and the acceleration of the target not being less than the preset linear velocity threshold and the preset acceleration threshold, respectively.

6. The method of claim 2 , further comprising, before fusing the body detection result and the leg detection result to obtain the fusion result, and controlling the robot to follow the target based on the fusion result, checking the body detection result and the leg detection result;
wherein fusing the body detection result and the leg detection result to obtain the fusion result comprises:
fusing the body detection result and the leg detection result after they have been checked; and

wherein the leg detection result comprises position information of a plurality of objects, and checking the leg detection result comprises:
filtering out one or more of the plurality of objects outside a preset second position range according to the leg detection result, wherein the preset second position range is a best range of operation of the sensor;
determining a distance between a remaining object of the plurality of objects and the camera, wherein the distance is determined by: calculating positions of shoulder centers in a body skeleton of the remaining object on a color image after the body skeleton of the remaining object in the color image taken by the camera is recognized using the pre-trained neural network model, obtaining depth information through a depth channel of the camera, and determining the distance between the remaining object and the camera according to the depth information; and
determining that the target has been detected in response to the distance between the remaining object and the camera being within a preset third position range, wherein the preset third position is an approximate radius of a circle surrounding a standing human.

7. The method of claim 1 , wherein fusing the body detection result and the leg detection result to obtain the fusion result and controlling the robot to follow the target based on the fusion result comprises:
controlling, in response to only one of the body detection result and the leg detection result showing that the target is detected, the robot to follow the target according to the one of the body detection result and the leg detection result;
controlling, in response to both the body detection result and the leg detection result showing that the target is detected, and a dis

Assignees

• Ubtech North America Res And Development Center Corp
• Ubtech Robotics Corp Ltd
• Ubkang Qingdao Tech Co Ltd

Inventors

• Guo Dejun
• Chou Ting-Shuo
• Shen Yang
• Tan Huan

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G05D1/2437

  Extracting relative motion information · CPC title

• G05D1/686

  Maintaining a relative position with respect to moving targets, e.g. following animals or humans (for pointing payloads towards targets G05D1/689; involving controlling the position or course of two or more vehicles G05D1/69) · CPC title

• G06V40/172

  Classification, e.g. identification · CPC title

• G06V40/103

  Static body considered as a whole, e.g. static pedestrian or occupant recognition · CPC title