Systems and methods for machine vision robotic processing

Patent metadata

Field	Value
Publication number	US-2023196599-A1
Application number	US-202217963156-A
Country	US
Kind code	A1
Filing date	Oct 10, 2022
Priority date	Oct 8, 2021
Publication date	Jun 22, 2023
Grant date	—

Abstract

Systems, methods, and media for machine vision-guided robotic loading are provided. Such systems and methods can allow for more accurate object detection so as to allow for robotic picking or loading of an object from a pile or group of objects. In some embodiments, a camera is controlled to capture a background image of the object or group of objects. One or more illumination sources (such as, e.g., laser illuminators) are controlled to project immunization toward the object or group of objects within the field-of-view of the camera. The camera may be controlled to capture a scan image comprising the projected illumination. Based on the scan image, a depth map can be determined, which is provided to a machine learning model along with the background image. The output of the machine learning model can then provide information associated with one or more objects, such as location information, orientation information, and the like.

First claim

1 . A system for detecting an object on a surface, comprising:
a camera, wherein the camera is oriented with the surface in a field-of-view of the camera; a plurality of aimable illumination sources configured to project an illumination toward the surface within the field-of-view of the camera; and a controller configured to:
control the camera to capture a background image of the object on the surface;
control the plurality of aimable illumination sources to project illumination toward the surface;
control the camera to capture a scan image comprising the projected illumination;
determine a depth map based on the scan image;
provide the depth map and the background image to a machine learning model; and
receive an output from the machine learning model, wherein the output comprises information associated with the object.

2 . The system of claim 1 , wherein each of the aimable illumination sources comprises a line laser.

3 . The system of claim 2 , wherein each of the aimable illumination sources further comprises:
a motor; and a mirror coupled to the motor, wherein:
the mirror is positioned to reflect illumination from the line laser; and
the motor is controllable to rotate the mirror based on a received control input.

4 . The system of claim 1 , wherein the plurality of aimable illumination sources comprise a first aimable illumination source and a second aimable illumination source, and wherein the first aimable illumination source and the second aimable illumination source have different spectrums.

5 . The system of claim 1 , wherein the controller is further configured to:
determine an object scan set at each of a plurality of scan steps, wherein the controller controls positioning of the projected illumination of each aimable illumination source based on the scan step; and determine the depth map based on the object scan set.

6 . The system of claim 5 , wherein the controller is further configured to:
determine a baseline scan set for each of the plurality of aimable illumination sources; and wherein determining the depth map based on the object scan set comprises subtracting the baseline scan set from the object scan set.

7 . The system of claim 1 , wherein the controller is further configured to control a robotic picking device to acquire the object according to the received output of the machine learning model.

8 . The system of claim 1 , wherein the controller is further configured to:
determine, based on the received output of the machine learning model, a first object among a plurality of objects to prioritize; and provide a first signal to the robotic picking device to aid in acquiring the first object.

9 . The system of claim 8 , wherein the controller is further configured to, after the first object has been removed from the surface:
provide a second depth map and a second background image to the machine learning model; receive a second output from the machine learning model, and based on the second output, determine a second object among the plurality of objects to prioritize; and provide a second signal to the robotic picking device to aid in acquiring the second object.

10 . The system of claim 1 , wherein the machine learning model is configured to:
identify the object; and wherein the information associated with the object comprises at least one of a mask, key point, class, and bounding box for the identified object.

11 . A method for detecting an object, comprising:
controlling a camera to capture a background image of the object on the surface, wherein the camera is oriented with the surface in a field-of-view of the camera; controlling a plurality of aimable illumination sources to project illumination toward the surface and within the field-of-view of the camera; controlling the camera to capture a scan image comprising the projected illumination; determining a depth map based on the scan image; providing the depth map and the background image to a machine learning model; and receiving an output from the machine learning model, wherein the output comprises information associated with the object.

12 . The method of claim 11 , wherein each of the aimable illumination sources comprises a line laser.

13 . The method of claim 12 , wherein each of the aimable illumination sources further comprises:
a motor; and a mirror coupled to the motor and positioned to reflect illumination from the line laser; and wherein:
controlling the plurality of aimable illumination sources comprises providing a control input to the motor of each aimable illumination source.

14 . The method of claim 11 , wherein the plurality of aimable illumination sources comprise a first aimable illumination source and a second aimable illumination source, and wherein the first aimable illumination source and the second aimable illumination source have different spectrums.

15 . The method of claim 11 , wherein the method further comprises:
determining an object scan set at each of a plurality of scan steps, and for each of the plurality of aimable illumination sources, controlling the aimable illumination source to project illumination from the aimable illumination source at a position determined according to the scan step; and determining the depth map based on the object scan set.

16 . The method of claim 15 , wherein the method further comprises:
determining a baseline scan set for each of the plurality of aimable illumination sources; and wherein determining the depth map based on the object scan set comprises subtracting the baseline scan set from the object scan set.

17 . The method of claim 11 , wherein the method further comprises generating a control signal for a robotic picking device to acquire the object according to the received output of the machine learning model.

18 . The method of claim 11 , wherein the method further comprises:
determining, based on the received output of the machine learning model, a first object among a plurality of objects to prioritize; and wherein controlling the robotic picking device to acquire the first object comprises controlling the robotic picking device to acquire the first object.

19 . The method of claim 18 , wherein the method further comprises determining a second object among the plurality of objects to prioritize, after the first object has been removed from the surface.

20 . The method of claim 11 , wherein the machine learning model is configured to:
identify the object; and wherein the information associated with the object comprises at least one of a mask, key point, class, and bounding box for the identified object.

Assignees

• Univ Maryland

Inventors

• Tao Yang
• Wang Dongyi
• Ali Mohamed Amr

Classifications

• B25J9/1697

  Vision controlled systems · CPC title

• G06V10/141 Primary

  Control of illumination · CPC title

• G06T7/593 Primary

  from stereo images · CPC title

• G05B2219/45063

  Pick and place manipulator · CPC title

• G05B2219/40053

  Pick 3-D object from pile of objects · CPC title