Robotic system with error detection and dynamic packing mechanism

Patent metadata

Field	Value
Publication number	US-10953549-B2
Application number	US-202016802451-A
Country	US
Kind code	B2
Filing date	Feb 26, 2020
Priority date	May 31, 2019
Publication date	Mar 23, 2021
Grant date	Mar 23, 2021

Abstract

A method for operating a robotic system includes determining a discretized object model based on source sensor data; comparing the discretized object model to a packing plan or to master data; determining a discretized platform model based on destination sensor data; determining height measures based on the destination sensor data; comparing the discretized platform model and/or the height measures to an expected platform model and/or expected height measures; and determining one or more errors by (i) determining at least one source matching error by identifying one or more disparities between (a) the discretized object model and (b) the packing plan or the master data or (ii) determining at least one destination matching error by identifying one or more disparities between (a) the discretized platform model or the height measures and (b) the expected platform model or the expected height measures, respectively.

First claim

We claim:

1. A method for operating a robotic system, the method comprising:
analyzing source sensor data representing a target object at or approaching a start location, wherein analyzing the source sensor data includes determining a discretized object model representing a physical dimension or shape of the target object in two dimensions (2D) according to unit pixels;
analyzing destination sensor data representing a placement area associated with a task location, wherein analyzing the destination sensor data includes—
determining a discretized platform model representing a physical dimension or shape of the task location in 2D according to further unit pixels, and
determining a height measure representing a maximum height within a portion of the placement area corresponding to a set of the further unit pixels;

determining an error, wherein determining the error includes:
(i) determining a source matching error by identifying a disparity between (a) the discretized object model and (b) a packing plan or master data, wherein the packing plan indicates placement locations and poses of objects at the task location, and wherein the master data includes descriptions of possible objects that are pre-registered with the robotic system, or
(ii) determining a destination matching error by identifying a disparity between (a) the discretized platform model or the height measure and (b) an expected platform model or an expected height measure, respectively; and

in response to determining the error, adjusting a placement location for the target object and/or another object.

2. The method of claim 1 , wherein:
analyzing the source sensor data includes comparing the discretized object model to the master data;
determining the error includes identifying a disparity between the discretized object model and the master data; and
the error includes a master data error representing the master data is wrong.

3. The method of claim 1 , wherein:
the packing plan specifies a sequence in which objects including the target object are to arrive at the start location;
analyzing the source sensor data includes comparing the discretized object model to the packing plan by comparing the discretized object model to a discretized object model specified by the sequence;
determining the error includes identifying a disparity between the discretized object model and the discretized object model specified by the sequence; and
the error includes an arrival sequence error representing that the target object arrived at the start location out of the sequence specified by the packing plan.

4. The method of claim 1 , wherein:
analyzing the destination sensor data includes comparing the discretized platform model to the expected platform model;
determining the error includes identifying a disparity between the discretized platform model and the expected platform model; and
the error includes a placement accessibility error, an unexpected placement error, and/or a placement area error representing that the placement area associated with the task location differs from an expected placement area associated with the task location.

5. The method of claim 4 , wherein:
determining the error further includes identifying a real-time packaging condition that poses a risk of collision;
identifying the real-time packaging condition includes determining a placement accessibility error representing a wall of a container, cage, or car track at the task location is not fully opened such that less than all of the placement area associated with the task location is accessible to the robotic system; and
the risk of collision is a risk of collision between the robotic system and the container, cage, or car track.

6. The method of claim 1 , wherein:
determining the error includes identifying a disparity between (a) the discretized platform model or the height measure and (b) the expected platform model or the expected height measure, respectively;
determining the error further includes identifying a real-time packaging condition that poses a risk of collision;
identifying the real-time packaging condition includes determining an unexpected placement error representing an object of previously placed objects at the task location shifted, fell, and/or was displaced; and
the risk of collision is a risk of collision between the robotic system and the object of the previously placed objects at the task location.

7. The method of claim 1 , wherein:
determining the error includes identifying a disparity between (a) the discretized platform model or the height measure and (b) the expected platform model or the expected height measure, respectively;
determining the error further includes identifying a real-time packaging condition that poses a risk of collision;
identifying the real-time packaging condition includes determining a placement area error representing an object of previously placed objects at the task location was misplaced; and
the risk of collision is a risk of collision between the robotic system and the object of the previously placed objects at the task location.

8. The method of claim 1 , wherein:
the expected platform model and/or the expected height measure are specified by the packing plan;
determining the error includes identifying a disparity between (a) the discretized platform model or the height measure and (b) the expected platform model or the expected height measure, respectively;
determining the error further includes identifying a real-time packaging condition that poses a risk of collision;
identifying the real-time packaging condition includes determining a placement area error representing an object not included in the packing plan is positioned at the task location; and
the risk of collision is a risk of collision between the robotic system and the object not included in the packing plan.

9. The method of claim 1 , wherein:
determining the error further includes identifying a real-time packaging condition that poses a risk of collision; and
identifying the real-time packaging condition includes determining errors in the following order:
determining a placement accessibility error representing a wall of a container, cage, or car track at the task location is not fully opened such that less than all of the placement area associated with the task location is accessible to the robotic system and the risk of collision is a risk of collision between the robotic system and the container, cage, or car track at the task location;
determining an unexpected placement error representing a first object of previously placed objects at the task location shifted, fell, and/or was displaced such that the risk of collision is a risk of collision between the robotic system and the first object of the previously placed objects at the task location;
determining a placement area error representing a second object of the previously placed objects at the task location was misplaced such that the risk of collision is a risk of collision between the robotic system and the second object of the previously placed objects at the task location; and
determining a placement area error representing a disparity between the height measure and the expected height measure such that the risk of collision is a risk of collision between the robotic system and objects contributing to the disparity between the height measure and the expected height measure, wherein the objects contributing to the disparity include (a) a third object of the previously placed objects, (b) the container, cage, or car track, and/or (c) an object not included in the packing plan but present at the task location.

10. The method of claim 1 , wherein:
determining

Assignees

• Mujin Inc

Inventors

• Diankov Rosen Nikolaev
• Kanunikov Denys

Classifications

• B25J9/161

  Hardware, e.g. neural networks, fuzzy logic, interfaces, processor · CPC title

• B25J9/1661

  characterised by task planning, object-oriented languages · CPC title

• B65G57/03

  from above · CPC title

• B65G57/22

  in layers each of predetermined arrangement · CPC title

• B25J9/1694

  characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion · CPC title