Object hand-over between robot and actor

Patent metadata

Field	Value
Publication number	US-10913151-B1
Application number	US-201916581134-A
Country	US
Kind code	B1
Filing date	Sep 24, 2019
Priority date	Jun 1, 2018
Publication date	Feb 9, 2021
Grant date	Feb 9, 2021

Abstract

An example robotic device may include an arm having a palm and fingers, a depth sensor disposed within the palm, and a control system. The control system may be configured to detect an indication to receive an object from an actor, and in response, cause the robotic device to enter a receive-object mode. When the robotic device is in the receive-object mode, the control system is further configured to: receive, from the depth sensor, depth data indicating a distance between the palm and the object; when the distance is greater than a first threshold and less than a second threshold, cause the arm to move towards the object; when the distance exceeds the second threshold, maintain the arm in a fixed position; and when the distance drops below the first threshold, cause the two or more fingers to close to grasp the object.

First claim

What is claimed is:

1. A robotic device comprising:
an arm comprising a palm and two or more fingers;
a first sensor;
an image sensor disposed within the palm; and
a control system configured to:
receive, from the first sensor, first data associated with an actor;
based on the first data, detect an indication to hand an object over from the robotic device to the actor, wherein the robotic device is holding the object between the two or more fingers; and
in response to detecting the indication to hand the object over from the robotic device to the actor, cause the robotic device to enter a give-object mode, wherein, when the robotic device is in the give-object mode, the control system is further configured to:
receive, from the image sensor, image data representing an environment around the palm, wherein the image data comprises (i) a first area containing the object held between the two or more fingers and (ii) a second area proximate to the object;
when the second area proximate to the object is vacant, cause the arm to move towards the actor; and
when a hand of the actor is detected in the second area, cause (i) the arm to stop and (ii) the two or more fingers to open to release the object into the hand of the actor.

2. The robotic device of claim 1 , wherein, when the robotic device is in the give-object mode, the control system is further configured to:
cause the two or more fingers to open to release the object into the hand of the actor in response to the hand of the actor being detected in the second area for at least a threshold period of time.

3. The robotic device of claim 1 , wherein, when the robotic device is in the give-object mode, the control system is further configured to:
determine, based on the image data, that the second area exceeds a size threshold such that the object obstructs no more than a threshold portion of a field of view of the image sensor; and
cause the arm to move towards the actor further based on determining that the second area exceeds the size threshold.

4. The robotic device of claim 1 , wherein the second area proximate to the object comprises at least one of, (i) an area below the object, (ii) an area above the object, or (iii) an area to a side of the object.

5. The robotic device of claim 1 , wherein, when the robotic device is in the give-object mode, the control system is further configured to:
detect the hand in the second area by comparing an intensity of at least a portion of pixels in the second area to a threshold intensity.

6. The robotic device of claim 1 , wherein, when the robotic device is in the give-object mode, the control system is further configured to:
determine a baseline intensity of pixels in the second area; and
detect the hand in the second area by comparing an intensity of at least a portion of pixels in the second area to the baseline intensity.

7. The robotic device of claim 1 , wherein the image data comprises a third area proximate to the object that is expected to be vacant, and wherein, when the robotic device is in the give-object mode, the control system is further configured to:
detect the hand in the second area by comparing an intensity of at least a portion of pixels in the second area to an intensity of at least a portion of pixels in the third area.

8. The robotic device of claim 1 , wherein the first sensor comprises an additional image sensor, and wherein, when the robotic device is in the give-object mode, the control system is further configured to:
determine that the second area is smaller than a threshold size due to the object occluding a field of view of the image sensor; and
in response to determining that the second area is smaller than the threshold size and based on additional image data received from the additional image sensor, (i) cause the arm to move towards the actor, (ii) cause the arm to stop, and (iii) cause the two or more fingers to open to release the object into the hand of the actor, wherein the additional image data represents the environment around the palm from a first perspective different from a second perspective of the image sensor disposed within the palm.

9. The robotic device of claim 1 , further comprising a depth sensor disposed within the palm, wherein, when the robotic device is in the give-object mode, the control system is further configured to:
receive, from the depth sensor, depth data representing (i) a first distance between the palm and physical features within the first area and (ii) a second distance between the palm and physical features in the second area;
when (i) the second area proximate to the object is vacant and (ii) the second distance is greater than the first distance, cause the arm to move towards the actor; and
when the hand of the actor is detected in the second area and the second distance is smaller than the first distance, cause the arm to stop and the two or more fingers to open to release the object into the hand of the actor.

10. The robotic device of claim 1 , wherein, when the robotic device is in the give-object mode, the control system is further configured to:
move the arm along a predetermined trajectory to gesture that the robotic device is in the give-object mode and is ready to hand the object over to the actor.

11. The robotic device of claim 1 , further comprising an additional image sensor, wherein, when the robotic device is in the give-object mode, the control system is further configured to:
receive, from the additional image sensor, additional image data representing a relative position between the arm and the hand;
based on the additional image data, determine a trajectory for moving the arm towards the hand; and
when the second area proximate to the object is vacant, causing the arm to move towards the actor along the trajectory.

12. The robotic device of claim 1 , wherein the first sensor comprises an additional image sensor, wherein the first data comprises additional image data, and wherein the control system is configured to detect the indication to hand the object over from the robotic device to the actor by:
identifying, based on the additional image data, a virtual skeleton representing the actor; and
detecting the indication to hand the object over from the robotic device to the actor based on a pose of the virtual skeleton reaching a predetermined configuration.

13. The robotic device of claim 1 , wherein the first sensor comprises a microphone communicatively connected to the control system, and wherein the control system is configured to detect the indication to hand the object over from the robotic device to the actor by:
detecting, by way of the microphone, an utterance by the actor; and
determining that the utterance indicates to hand the object over from robotic device to the actor.

14. The robotic device of claim 1 , wherein the first sensor comprises an additional image sensor, wherein the first data comprises additional image data, wherein the robotic device further comprises an audio speaker and a microphone, and wherein the control system is configured to detect the indication to hand the object over from the robotic device to the actor by:
identifying, based on the additional image data, a virtual skeleton representing the actor;
in response to a pose of the virtual skeleton reaching a predetermined configuration, causing the audio speaker to generate an utterance requesting reception of the object from the arm by the actor; and
detecting, by way of the microphone, an utterance by the actor indicating to hand the object over from the robotic device to the actor.

15. The robotic de

Assignees

• X Dev Llc

Inventors

• Hinkle Paige

Classifications

• G05B2219/39536

  Planning of hand motion, grasping · CPC title

• B25J9/1697 Primary

  Vision controlled systems · CPC title

• B25J9/1612 Primary

  characterised by the hand, wrist, grip control · CPC title

• G05B2219/40298

  Manipulator on vehicle, wheels, mobile · CPC title

• B25J9/161

  Hardware, e.g. neural networks, fuzzy logic, interfaces, processor · CPC title