System and method for semantically identifying one or more of an object and a location in a robotic environment

Patent metadata

Field	Value
Publication number	US-11835960-B2
Application number	US-201916259261-A
Country	US
Kind code	B2
Filing date	Jan 28, 2019
Priority date	Jan 28, 2019
Publication date	Dec 5, 2023
Grant date	Dec 5, 2023

Abstract

A system includes: a mobile robot comprising a sensor, the robot further comprising a computer, the robot operating in an environment; a server operably connected to the robot via a communication system, the server configured to manage the robot; a controller operably connected to the robot, the controller operably connected to the server, the controller configured to control the robot; and an object of interest marked with a marker at one or more of an approximate height and an approximate field of view of the sensor, the sensor generating data describing the object of interest, the computer configured to identify one or more of the object of interest and the location using one or more of a shape of the object of interest and an intensity of data describing the object of interest.

First claim

What is claimed is:

1. A system for semantically identifying an object of interest, comprising:
a mobile robot operating in an environment, the mobile robot comprising (i) a sensor configured to generate sensor data representing a portion of the environment within a field of view of the sensor, and (ii) a computer, the robot operating in the environment;
a server operably connected to the robot via a communication system, the server configured to manage the robot;
a controller operably connected to the robot, the controller operably connected to the server, the controller configured to control the robot; and
a marker disposed in the environment within a field of view of the sensor;
the computer configured to:
detect the marker from the sensor data;
based on the detected marker, determine an identity code associated with an object of interest outside the field of view of the sensor; and
based on the determined identity code, identify a location of the object of interest outside of the field of view of the sensor and cause the mobile robot to perform a safety action while the object of interest remains outside of the field of view of the sensor of the mobile robot.

2. The system of claim 1 , wherein the sensor comprises an infrared sensor.

3. The system of claim 1 , wherein the sensor comprises a laser sensor.

4. The system of claim 1 , wherein the sensor comprises a three-dimensional (3D) sensor.

5. The system of claim 1 , wherein the sensor comprises a three-dimensional (3D) camera configured to provide both two-dimensional (2D) spatial information and also information on depth of the object of interest.

6. The system of claim 1 , wherein the marker comprises a QR code.

7. The system of claim 6 , wherein the sensor includes a downward-facing camera, and wherein the QR code is visible to the downward-facing camera.

8. The system of claim 7 , wherein the system uses the marker to automatically calibrate the camera.

9. The system of claim 1 , wherein the object of interest is selected from the group consisting of a forklift, a box, a pallet, a second robot, a cart, a golf cart, a dock bay door, and an obstacle disposed outside of the field of view of the sensor.

10. A method for semantically identifying an object of interest in an environment of a mobile robot, comprising:
receiving, from a sensor of the mobile robot by a computer comprised in the mobile robot, sensor data representing a portion of the environment within a field of view of the sensor;
detecting, by the computer from the sensor data, a marker disposed in the environment within a field of view of the sensor and associated with an object of interest outside the field of view of the sensor;
based on the detected marker, determining, by the computer, an identity code associated with the identified object of interest outside the field of view of the sensor;
using the identity code, by the computer, determining a location of the object of interest;
sending, by the computer, the determined location to a server; and
performing, by the mobile robot, a safety action while the object of interest remains outside of the field of view of the sensor of the mobile robot.

11. A method for semantically identifying, using a first mobile robot, a location of a mis-localized second mobile robot in an environment of the first mobile robot, comprising:
retrieving, by a server operably connected to the first mobile robot, a last known location of the mis-localized second mobile robot, the second mobile robot having a marker;
ordering, by the server, the first mobile robot to look for the second robot in the last known location of the second mobile robot, the first mobile robot configured, in response to the ordering, to detect the marker in sensor data generated by a sensor of the first mobile robot, the sensor data representing a portion of the environment within a field of view of the sensor;
receiving, at the server from the first mobile robot, data encoded in the detected marker; and
generating, by the server, a high-confidence identification of the current location of the second mobile robot, using the data encoded in the marker detected by the first mobile robot.

12. The method of claim 11 , further comprising a step of:
using the determined current location, by the server, automatically placing the second mobile robot in a correct location in a map.

13. The method of claim 12 , further comprising a step, performed after the placing step, of instructing the first robot, by the server, to apply a barcode to a charge dock.

14. The method of claim 13 , further comprising a step, performed after the instructing step, of counting, by the server, charge cycles using the barcode.

Assignees

• Fetch Robotics Inc
• Zebra Tech Corp

Inventors

• Cairl Brian
• King Derek
• Elliott Sarah
• Henning Alex
• Wise Melonee
• Toris Russell

Classifications

• G05D1/0248 Primary

  in combination with a laser (lasers per se H01S) · CPC title

• B25J9/162

  Mobile manipulator, movable base with manipulator arm mounted on it · CPC title

• B25J9/1666

  Avoiding collision or forbidden zones · CPC title

• G05D1/0234 Primary

  using optical markers or beacons (optical beacons per se G01S1/70) · CPC title

• G05D1/0251

  extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision (stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593) · CPC title