System for presenting robotic data flows for application development

Patent metadata

Field	Value
Publication number	US-12318933-B2
Application number	US-202217711973-A
Country	US
Kind code	B2
Filing date	Apr 1, 2022
Priority date	Apr 1, 2021
Publication date	Jun 3, 2025
Grant date	Jun 3, 2025

Abstract

A robot system includes a first computing device and a second computing device. The first computing device is configured to control operation of the robot based on data flows received from a plurality of sensors of the robot and the second computing system is configured to receive and process at least some of the data flows concurrently while the first computing system controls operation of the robot.

First claim

What is claimed is:

1. A system, comprising:
a robot having a first computing system configured to control operation of the robot based on data flows received from a plurality of sensors of the robot, the plurality of sensors of the robot including a camera, a servo, and a tactile sensor, the first computing system being onboard the robot and having a first network interface, the first computing system exposing an application program interface (API) by which at least some of the data flows are consumable by other computing devices; and
a second computing system having a second network interface and configured to both receive, via the API and the second network interface, at least some of the data flows and process the at least some of the data flows concurrently while the first computing system controls operation of the robot, wherein the second computing system:
applies labels to at least some of the data flows; and
provides labeled data flows to a reinforcement learning policy, wherein the reinforcement learning policy generates, based on the labeled data flows, instructions for performance of one or more actions by the robot;
and wherein the first computing system onboard the robot receives the instructions for the performance of one or more actions by the robot and executes the instructions to cause the robot to perform one or more actions.

2. The system of claim 1 , wherein the second computing system is configured to provide a dashboard representative of sensor data from the data flow.

3. The system of claim 1 , wherein the robot is a first robot, and the second computing system is configured to process and transmit data flows to one or more robots in a fleet including the first robot.

4. The system of claim 1 , wherein the second computing system is configured to receive an inbound connection to remotely control the robot.

5. The system of claim 1 , wherein the second computing system is configured to provide offline processing of the data flow while the first computing system is inactive or processing other data.

6. The system of claim 1 , wherein the second computing system is configured to determine whether a time slice of the data flows satisfy a defensive sampling criteria and, in response to determining that the time slice of the data flows do not satisfy the defensive sampling criteria determining to not advance to another process or to discard the time slice.

7. The system of claim 1 , wherein the second computing system is configured to provide at least some of the data flows to a process configured to perform a computer vision task corresponding to a mechanical task being performed by the robot.

8. The system of claim 7 , wherein the computer vision task is scanning an optical code on a workpiece being manipulated by the robot in the mechanical task.

9. The system of claim 1 , wherein the second computing system is configured to provide at least some of the data flows to a process configured to perform a statistical process control task corresponding to a mechanical task being performed by the robot.

10. The system of claim 1 , wherein the second computing system is configured to provide at least some of the data flows to a process configured to perform a manufacturing traceability task corresponding to a mechanical task being performed by the robot.

11. The system of claim 1 , wherein the second computing system is configured to provide, based on the data flows, image data by which an augmented reality display is rendered operative to guide a user manually operating the robot.

12. The system of claim 11 , wherein the augmented reality display is configured to display at least one of the following:
singularity points of the robot to be avoided for path planning,
optimal paths from a current state to an expected target state of the robot,
predicted next movements based on a current state of the robot of a partially or fully trained model by which the robot is controlled, or
instructions corresponding to a next step in a process being performed by the robot.

13. The system of claim 11 , wherein the augmented reality display is configured to display each of the following:
singularity points of the robot to be avoided for path planning,
optimal paths from a current state to an expected target state of the robot,
predicted next movements based on a current state of the robot of a partially or fully trained model by which the robot is controlled, and
instructions corresponding to a next step in a process being performed by the robot.

14. The system of claim 1 , wherein the second computing system comprises a cluster of computing devices configured to collectively execute a distributed real-time, complex event processing framework that processes the data flows.

15. The system of claim 1 , wherein the second computing system is configured to:
provide at least some of the data flows to a process configured to detect model suitability to a mechanical task being performed by the robot,
provide at least some of the data flows to a process configured to detect model suitability of a model by which the robot is controlled,
provide at least some of the data flows to a process configured to compare planned versus executed paths of the robot, or
provide at least some of the data flows to a process configured to detect noise in movements of the robot.

16. The system of claim 15 , wherein the second computing system is configured to:
provide at least some of the data flows to a first process configured to detect model suitability to a mechanical task being performed by the robot,
provide at least some of the data flows to a second process configured to detect model suitability of a model by which the robot is controlled,
provide at least some of the data flows to a third process configured to compare planned versus executed paths of the robot, and
provide at least some of the data flows to a fourth process configured to detect noise in movements of the robot.

17. The system of claim 1 , wherein the second computing system is configured to perform means for providing an auxiliary application.

18. A method, comprising:
controlling a robot with a first computing system configured to control operation of the robot based on data flows received from a plurality of sensors of the robot, the plurality of sensors of the robot including a camera, a servo, and a tactile sensor, the first computing system being onboard the robot and having a first network interface, the first computing system exposing an application program interface (API) by which at least some of the data flows are consumable by other computing devices;
receiving, by a second computing system having a second network interface, via the API and the second network interface, at least some of the data flows and processing, by the second computing system, the at least some of the data flows concurrently while the first computing system controls operation of the robot;
applying, by the second computing system, labels to at least some of the data flows;
providing, by the second computing system, labeled data flows to a reinforcement learning policy, wherein the reinforcement learning policy generates, based on the labeled data flows, instructions for performance of one or more actions by the robot;
receiving, by the first computing system, the instructions for the performance of one or more actions by the robot; and
executing, by the first computing system, the instructions to cause the robot to perform one or more actions.

19. The system of claim 1 wherein

Assignees

• Sanctuary Cognitive Systems Corp

Inventors

• Wales Carolyn
• Cianci Chris
• Kranski Jeff
• Kaehler Adrian

Classifications

• B25J9/1689 Primary

  Teleoperation · CPC title

• B25J19/02

  Sensing devices · CPC title

• B25J9/1664

  characterised by motion, path, trajectory planning · CPC title

• B25J9/1697

  Vision controlled systems · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title