System and method for predicting robotic tasks with deep learning

Patent metadata

Field	Value
Publication number	US-11203116-B2
Application number	US-201916530344-A
Country	US
Kind code	B2
Filing date	Aug 2, 2019
Priority date	Aug 2, 2019
Publication date	Dec 21, 2021
Grant date	Dec 21, 2021

Abstract

A computing system is provided for training one or more machine learning models to perform at least a portion of a robotic task of a physical robotic system by monitoring a model-based control algorithm associated with the physical robotic system perform at least a portion of the robotic task. One or more robotic task predictions may be defined, via the one or more machine learning models, based upon, at least in part, the training of the one or more machine learning models. The one or more robotic task predictions may be provided to the model-based control algorithm associated with the physical robotic system. The robotic task may be performed, via the model-based control algorithm associated with the robotic system, on the physical robotic system based upon, at least in part, the one or more robotic task predictions defined by the one or more machine learning models.

First claim

What is claimed is:

1. A computing system including a processor and memory configured to perform operations comprising:
training one or more apprentice machine learning models to perform at least a portion of a robotic task of a physical robotic system by monitoring a model-based control algorithm associated with the physical robotic system perform at least a portion of the robotic task;
defining, via the one or more apprentice machine learning models, one or more robotic task predictions based upon, at least in part, the training of the one or more apprentice machine learning models;
wherein the task predictions comprise at least one of:
identification of a 3D pose or location of an object;
identification of an object;
location of one or more pickable regions within a bin; or
selection of a region of interest within a bin;

providing the one or more robotic task predictions to the model-based control algorithm associated with the physical robotic system; and
performing, via the model-based control algorithm associated with the robotic system, the robotic task on the physical robotic system based upon, at least in part, the one or more robotic task predictions defined by the one or more apprentice machine learning models.

2. The computing system of claim 1 , wherein the robotic task includes picking at least one object from a bin.

3. The computing system of claim 2 , wherein defining, via the one or more machine learning models, the one or more robotic task predictions includes:
identifying, via a first apprentice machine learning model, a location of one or more pickable regions in at least a portion of a bin.

4. The computing system of claim 3 , wherein providing the one or more robotic task predictions to the model-based control algorithm includes:
providing the location of the one or more pickable regions in at least a portion of the bin to the model-based control algorithm.

5. The computing system of claim 4 , wherein performing, via the model-based control algorithm associated with the robotic system, the robotic task on the physical robotic system based upon, at least in part, the one or more robotic task predictions defined by the one or more apprentice machine learning models includes:
selecting, via the model-based control algorithm, a region of interest based upon, at least in part, the location of the one or more pickable regions in at least a portion of the bin.

6. The computing system of claim 5 , wherein training the one or more apprentice machine learning models to perform the at least a portion of the robotic task of the physical robotic system by monitoring the model-based control algorithm associated with the physical robotic system perform the at least a portion of the robotic task includes:
identifying, via the model-based control algorithm, one or more objects within the selected region of interest, thus defining one or more identified objects;
scoring, via the model-based control algorithm, the one or more identified objects using a model fitting criteria, thus defining one or more ranked objects;
performing one or more simulations to determine which of the one or more ranked objects can be picked by the physical robotic system, thus defining one or more pickable objects.

7. The computing system of claim 6 , further comprising:
receiving a pick validation from the model-based control algorithm indicating whether an identified object is valid based upon, at least in part, the one or more simulations or invalid based upon, at least in part, the scoring of the one or more identified objects;
generating a training pair including a scanned image of at least a portion of the bin and a corresponding heatmap indicating a location of the identified object; and
updating the first apprentice machine learning model based upon, at least in part, the training pair.

8. The computing system of claim 7 , wherein the location of the one or more pickable regions is provided to the model-based control algorithm after the first apprentice machine learning model is updated with a predetermined number of training pairs.

9. The computing system of claim 2 , wherein defining, via the one or more apprentice machine learning models, the one or more robotic task predictions includes:
defining, via a second apprentice machine learning model, one or more object locations in at least a portion of the bin based upon, at least in part, a scan of at least a portion of the bin.

10. The computing system of claim 9 , wherein providing the one or more robotic task predictions to the model-based control algorithm includes:
providing the one or more object locations in at least a portion of the bin to the model-based control algorithm.

11. The computing system of claim 10 , wherein performing, via the model-based control algorithm associated with the robotic system, the robotic task on the physical robotic system based upon, at least in part, the one or more robotic task predictions defined by the one or more apprentice machine learning models includes:
identifying, via the model-based control algorithm, one or more objects within a selected region of interest based upon, at least in part, the one or more object locations, thus defining one or more apprentice machine learning model-identified objects.

12. The computing system of claim 9 , wherein the one or more object locations in at least a portion of the bin includes a list of bounding boxes and a binary mask associated with the at least one object.

13. The computing system of claim 8 , further comprising:
identifying, via the model-based control algorithm, one or more objects within the selected region of interest, thus defining one or more identified objects.

14. The computing system of claim 13 , further comprising:
updating the second apprentice machine learning model based upon, at least in part, the one or more identified objects.

15. The computing system of claim 14 , wherein defining, via the one or more apprentice machine learning models, the one or more robotic task predictions includes:
generating, via a third apprentice machine learning model, a three-dimensional pose associated with the at least one object based upon, at least in part, a scan of the at least a portion of the bin and the one or more object locations in at least a portion of the bin.

16. The computing system of claim 15 , further comprising:
generating, via the third apprentice machine learning model, a validation score for each three-dimensional pose associated with the at least one object.

17. The computing system of claim 16 , wherein providing the one or more object predictions to the model-based control algorithm includes:
providing the three-dimensional pose associated with the at least one object and the validation score for each three-dimensional pose associated with the at least one object to the model-based control algorithm.

18. The computing system of claim 17 , wherein performing, via the model-based control algorithm associated with the robotic system, the robotic task on the physical robotic system based upon, at least in part, the one or more robotic task predictions defined by the one or more apprentice machine learning models includes:
scoring, via the model-based control algorithm, the three dimensional pose associated with each object of the at least one object; and
performing one or more simulations to determine which three dimensional pose associated with the at least one object can be picked by the physical robotic system.

19. The computing system of claim 1

Assignees

• Teradyne Inc

Inventors

• Demirdjian David
• Truebenbach Eric Lenhart

Classifications

• B25J9/163 Primary

  learning, adaptive, model based, rule based expert control · CPC title

• G05B2219/41054

  Using neural network techniques · CPC title

• G05B2219/50391

  Robot · CPC title

• G05B13/048

  using a predictor · CPC title