Collision avoidance based on vision data, geometric data and physics engine evaluation

Patent metadata

Field	Value
Publication number	US-12486125-B2
Application number	US-202217837797-A
Country	US
Kind code	B2
Filing date	Jun 10, 2022
Priority date	Jun 16, 2021
Publication date	Dec 2, 2025
Grant date	Dec 2, 2025

Abstract

A robotic system is disclosed. The system includes a communication interface configured to receive, from one or more sensors deployed in a workspace, sensor data indicative of a current state of the workspace, the workspace comprising a pallet or other receptacle and a plurality of items stacked on or in the receptacle. The system includes one or more processors that use a geometric model based at least in part on past item placements in combination with the sensor data to estimate a state of the pallet or other receptacle and one or more items stacked on or in the pallet or other receptacle, and use the estimated state to generate or update a plan to control a robotic arm to place a next item on or in, or remove a next item from, the pallet or other receptacle in a manner that avoids having the next item collide with any other item stacked on or in the pallet or other receptacle.

First claim

What is claimed is:

1 . A robotic system, comprising:
a communication interface configured to receive, from one or more sensors deployed in a workspace, sensor data indicative of a current state of the workspace, the workspace comprising a pallet or other receptacle and a plurality of items stacked on or in the receptacle; and one or more processors coupled to the communication interface and configured to:
use a geometric model based at least in part on past item placements in combination with the sensor data to estimate a state of the pallet or other receptacle and one or more items stacked on or in the pallet or other receptacle; and
use the estimated state to generate or update a plan to control a robotic arm to place a next item on or in the pallet or other receptacle in a manner that avoids having the next item collide with any other item stacked on or in the pallet or other receptacle, wherein generating or updating the plan includes simulating placement of the next item based at least in part on querying a placement model, wherein the placement model has a predefined noise profile that simulates noise corresponding to a difference between placement according to the geometric model and an actual placement.

2 . The robotic system of claim 1 , wherein generating or updating the plan comprises:
computing a trajectory along which the next item is to be moved, the trajectory being computed based on the estimated state to avoid collision between the next item or the robotic arm with (i) any other item stacked on or in the pallet or other receptacle, and (ii) another object in the workspace.

3 . The robotic system of claim 1 , wherein the one or more processors are further configured to:
detect an object in the workspace; in response to detecting the object, query a state estimator to obtain the estimated state; and generate or update a plan to control a robotic arm to move the item in a manner that avoids a collision between the object and the item or the robotic arm.

4 . The robotic system of claim 3 , wherein generating or updating the plan to control the robotic arm to move the item in the manner that avoids a collision between the object and the item or the robotic arm comprises:
determining a trajectory along which the item is to be moved from a current location to a destination location to avoid the collision with the object.

5 . The robotic system of claim 1 , wherein the one or more processors are further configured to:
determine an uncertainty of the estimated state; determine if the uncertainty of the estimated state exceeds an uncertainty threshold; and in response to determining that the uncertainty of the estimated state exceeds the uncertainty threshold, prompt human intervention.

6 . The robotic system of claim 5 , wherein to prompt human intervention includes communicating an alert to a client system being used by a user.

7 . The robotic system of claim 5 , wherein the uncertainty of the estimated state is based on a difference between the geometric model and sensor data obtained by one or more sensors in the workspace.

8 . The robotic system of claim 7 , wherein uncertainty is deemed to exceed the uncertainty threshold in response to a determination that a difference between the geometric model and the sensor data exceeds a predefined difference threshold.

9 . The robotic system of claim 1 , wherein the plan is determined based at least in part on performing a simulation of a placement of the next item.

10 . The robotic system of claim 9 , wherein the performing the simulation of the placement of the next item comprises invoking a physics engine to simulate a stack including (i) the plurality of items, and (ii) the next item placed in accordance with the placement.

11 . The robotic system of claim 10 , wherein the placement is selected from a set of possible placements based at least in part on a corresponding score with respect to a predefined scoring function.

12 . The robotic system of claim 11 , wherein the score for the placement is based at least in part on a stability of a simulated stack of items resulting from the placement being simulated.

13 . The robotic system of claim 1 , wherein using the estimated state to generate the plan to control the robotic arm to place a next item comprises:
determining a three-dimensional topography of a stack of the plurality of items; determining one or more attributes of the next item; and determine a trajectory along which the robotic arm is to move the next item to place the next item on the stack, the trajectory being determined based at least in part on (i) the three-dimensional topography of the stack, and (ii) the one or more attributes of the next item.

14 . The robotic system of claim 13 , wherein the trajectory along which the robotic arm is to move the next item is based at least in part on dimensions of the next item and a strategy of grasping the next item with the robotic arm from a top of the next item.

15 . The robotic system of claim 13 , wherein the trajectory is further determined based on a predefined cost associated with moving the next item along the trajectory from a current location to a destination location.

16 . The robotic system of claim 13 , wherein the trajectory is further determined based on selection of a corresponding placement of the next item based on a score of the placement with respect to a predefined scoring function.

17 . The robotic system of claim 13 , wherein the trajectory along which the robotic is to move the next item is determined based at least in part on a determination that the next item being moved or the robotic arm moving the next item does not intersect with any point(s) in space occupied by another item or object in the workspace.

18 . The robotic system of claim 17 , wherein the determination that the next item being moved or the robotic arm moving the next item does not intersect with any point(s) in space occupied by another item or object in the workspace is performed based on a three-dimensional representation of the next item and workspace.

19 . A method to control a robot, comprising:
receiving, from one or more sensors deployed in a workspace, sensor data indicative of a current state of the workspace, the workspace comprising a pallet or other receptacle and a plurality of items stacked on or in the receptacle; using, by one or more processors, a geometric model based at least in part on past item placements in combination with the sensor data to estimate a state of the pallet or other receptacle and one or more items stacked on or in the pallet or other receptacle; and using the estimated state to generate or update a plan to control a robotic arm to place a next item on or in the pallet or other receptacle in a manner that avoids having the next item collide with any other item stacked on or in the pallet or other receptacle, wherein generating or updating the plan includes simulating placement of the next item based at least in part on querying a placement model, wherein the placement model has a predefined noise profile that simulates noise corresponding to a difference between placement according to the geometric model and an actual placement.

20 . A computer program product to control a robot, the computer program product being embodied in a non-transitory computer readable medium and comprising computer instructions for:
receiving, from one or more sensors deployed in a workspace, sensor data indicative of a current state of the wo

Assignees

• Dexterity Inc

Inventors

• Pidaparthi Rohit Arka
• Clary William Arthur
• Abhyankar Neeraja
• Kuck Jonathan
• Pottayil Ben Varkey Benjamin
• Chavez Kevin Jose
• Kumar Shitij
• Moreno Robert
• Beladiya Arth
• Perez Salvador
• Baek Jeesu

Classifications

• B65G57/22

  in layers each of predetermined arrangement · CPC title

• B25J9/1664

  characterised by motion, path, trajectory planning · CPC title

• B65G2203/0233

  Position of the article · CPC title

• B65G2203/042

  Sensors · CPC title

• B65G57/03 Primary

  from above · CPC title