State estimation using geometric data and vision system for palletizing

Patent metadata

Field	Value
Publication number	US-12214975-B2
Application number	US-202217837775-A
Country	US
Kind code	B2
Filing date	Jun 10, 2022
Priority date	Jun 16, 2021
Publication date	Feb 4, 2025
Grant date	Feb 4, 2025

Abstract

A robotic system is disclosed. The system includes a communication interface that receives, from a sensor(s) deployed in a workspace, sensor data indicative of a current state of the workspace, the workspace comprising a pallet or other receptacle and a plurality of items stacked on or in the receptacle. The system includes one or more processors that control a robotic arm to place a first set of items on or in, or remove the first set of items from, the pallet or other receptacle, update a geometric model based on the first set of items placed on or in a receptacle, use the geometric model in combination with the sensor data to estimate a stack of one or more items on or in the receptacle, and use the estimated state to generate or update a plan to control the robotic arm to place a second set of items.

First claim

What is claimed is:

1. A robotic system, comprising:
a communication interface configured to receive, from one or more sensors deployed in a workspace, sensor data indicative of a current state of the workspace, the workspace comprising a pallet or other receptacle and a plurality of items stacked on the pallet or in the other receptacle; and
one or more processors coupled to the communication interface and configured to:
control a robotic arm to place a first set of items on or in, or remove the first set of items from, the pallet or the other receptacle;
update a geometric model based on the first set of items placed on or in, or removed from, the pallet or ether the other receptacle;
use the geometric model in combination with the sensor data to estimate a state of the pallet or the other receptacle and one or more items stacked on the pallet or in the other receptacle; and
use the estimated state to generate or update a plan to control the robotic arm to place a second set of items on or in, or remove the second set of items from, the pallet or the other receptacle, wherein the estimated state of the pallet or other receptacle and the one or more items stacked on the pallet or in the other receptacle is determined based at least in part on an interpolation between the sensor data and the geometric model.

2. The robotic system of claim 1 , wherein the geometric model reflects respective attributes of one or more of the first set of items and the second set of items.

3. The robotic system of claim 2 , wherein the geometric model reflects respective weights of items comprised in the one or more of the first set of items and the second set of items.

4. The robotic system of claim 1 , wherein the geometric model reflects a respective rigidity compressibility of items comprised in one or more of the first set of items and the second set of items.

5. The robotic system of claim 1 , wherein the geometric model reflects a respective strength of packaging of items comprised in one or more of the first set of items and the second set of items.

6. The robotic system of claim 1 , wherein the geometric model includes a distribution of weight across a plurality of items comprised in one or more of the first set of items and the second set of items.

7. The robotic system of claim 1 , wherein the geometric model comprises an expected stability of the one or more items stacked on the pallet or in the other receptacle.

8. The robotic system of claim 7 , wherein the expected stability is based at least in part on a set of locations at which the robotic system believes each item was placed, and one or more attributes of each item stacked on the pallet or in the other receptacle.

9. The robotic system of claim 1 , wherein the geometric model comprises an expected stability of the one or more items stacked on the pallet or in the other receptacle and one or more simulated items for which simulation of a stacking of the one or more simulated items is performed.

10. The robotic system of claim 1 , wherein the one or more sensors include image sensors.

11. The robotic system of claim 10 , wherein the image sensors comprise a 3D camera.

12. The robotic system of claim 1 , wherein a point cloud is determined based at least in part on the sensor data.

13. The robotic system of claim 1 , wherein a location of each item is determined based on an interpolation between (i) a location of the respective item according to the sensor data, and (ii) a location of the respective item according to the geometric model.

14. The robotic system of claim 1 , wherein the estimated state of the pallet or the other receptacle and the one or more items stacked on the pallet or in the other receptacle is determined based at least in part on a midpoint between an item location of a first item according to the sensor data and the item location of the first item according to the geometric model.

15. The robotic system of claim 1 , wherein the one or more processors are further configured to:
use the geometric model to update the current state of the workspace according to the sensor data in response to a determination that the current state of the workspace according to the sensor data comprises an anomaly.

16. The robotic system of claim 15 , wherein the anomaly is caused by one or more of (i) a sensor of the one or more sensors being blocked or obscured from view with respect to a particular point in the workspace, (ii) a noise or gap in the sensor data that causes a gap in the current state of the workspace.

17. The robotic system of claim 1 , wherein the one or more processors are further configured to:
determine that a difference between a state expected based on the geometric model and a state observed via the one or more sensors exceed a threshold.

18. The robotic system of claim 17 , wherein in response to a determination that the difference exceeds the threshold, communicate an alert to a user to confirm that a stack of the one or more items on the pallet or in the other receptacle is not problematic.

19. The robotic system of claim 1 , wherein the one or more processors comprise:
a first subset of the one or more processors that is configured to:
generate or update the plan to control the robotic arm to place the second set of items on or in, or remove the second set of items from, the pallet or other receptacle, wherein:
the plan to control the robotic arm is generated or updated based at least in part on the estimated state; and

a second subset of the one or more processors that is configured to:
update the geometric model and to determine the estimate of the state of the pallet or the other receptacle and the one or more items stacked on the pallet or in the other receptacle based at least in part on the geometric model in combination with the sensor data.

20. The robotic system of claim 19 , wherein the second subset of the one or more processors is remote from the workspace.

21. The robotic system of claim 1 , wherein:
the first set of items comprises N items; and
N is a positive integer that is dynamically determined.

22. The robotic system of claim 21 , wherein the state of the one or more items stacked on or in the pallet or the other receptacle is estimated after the N items are stacked.

23. The robotic system of claim 22 , wherein the state is estimated before the N items are stacked in response to a determination that an irregularly shaped item was placed.

24. The robotic system of claim 1 , wherein:
the second set of items corresponds to a next M items on a source to be placed on the pallet or receptacle; and
M is a positive integer.

25. The robotic system of claim 24 , wherein the source is a conveyor.

26. A method to control a robot, comprising:
receiving, from one or more sensors deployed in a workspace, sensor data indicative of a current state of the workspace, the workspace comprising a pallet or other receptacle and a plurality of items stacked on the pallet or in the other receptacle;
controlling, by one or more processors, a robotic arm to place a first set of items on or in, or remove the first set of items from, the pallet or the other receptacle;
updating a geometric model based on the first set of items placed on or in, or removed from, the pallet or the other receptacle;
using the geometric model in combination with the sensor data to estimate a state of the pallet or the other recep

Assignees

• Dexterity Inc

Inventors

• Pidaparthi Rohit Arka
• Clary William Arthur
• Abhyankar Neeraja
• Kuck Jonathan
• Pottayil Ben Varkey Benjamin
• Chavez Kevin Jose
• Kumar Shitij

Classifications

• B25J9/1697

  Vision controlled systems · CPC title

• B25J9/1661

  characterised by task planning, object-oriented languages · CPC title

• B25J9/1671

  characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems · CPC title

• G05B2219/45063

  Pick and place manipulator · CPC title

• G05B2219/40607

  Fixed camera to observe workspace, object, workpiece, global · CPC title