Systems and methods for camera calibration with a fiducial of unknown position on an articulated arm of a programmable motion device

Patent metadata

Field	Value
Publication number	US-11826918-B2
Application number	US-202117167648-A
Country	US
Kind code	B2
Filing date	Feb 4, 2021
Priority date	Feb 6, 2020
Publication date	Nov 28, 2023
Grant date	Nov 28, 2023

Abstract

A system is disclosed for providing extrinsic calibration of a camera to a relative working environment of a programmable motion device that includes an end-effector. The system includes a fiducial located at or near the end-effector, at least one camera system for viewing the fiducial as the programmable motion device moves in at least three degrees of freedom, and for capturing a plurality of images containing the fiducial, and a calibration system for analyzing the plurality of images to determine a fiducial location with respect to the camera to permit calibration of the camera with the programmable motion device.

First claim

What is claimed is:

1. A system for providing extrinsic calibration of a camera to a relative working environment of a programmable motion device that includes an end-effector, said system comprising:
a calibration unit attached to the end effector, the calibration unit including a plurality of fiducials;
at least one camera for viewing a fiducial among the plurality of fiducials of the calibration unit as the programmable motion device moves in at least three degrees of freedom, and for capturing a plurality of images containing the fiducial; and
a calibration processing system for analyzing the plurality of images to determine a fiducial location of the fiducial with respect to the at least one camera to permit calibration of the at least one camera with the programmable motion device,
the calibration unit including a coupling portion for coupling to the end-effector and a transverse portion that is transverse to the coupling portion, wherein the plurality of fiducials are positioned on mutually opposing sides of the transverse portion of the calibration unit, the plurality of fiducials including a light emitting diode (LED) on a top side or a bottom side of the transverse portion, wherein the transverse portion of the calibration unit further includes control circuitry for controlling the LED.

2. The system as claimed in claim 1 , wherein the LED provides intermittent visible illumination.

3. The system as claimed in claim 1 , wherein the LED provides intermittent infra-red illumination.

4. The system as claimed in claim 1 , wherein the plurality of fiducials further includes a retroreflective fiducial.

5. The system as claimed in claim 4 , wherein the retroreflective fiducial is at least partially spherically shaped.

6. The system as claimed in claim 1 , wherein the calibration system constructs a matrix for each of the plurality of images of the fiducial.

7. The system as claimed in claim 6 , wherein the calibration system concatenates extrinsic parameters of the camera to the programmable motion device.

8. The system as claimed in claim 7 , wherein the calibration system employs non-linear least square optimization.

9. The system as claimed in claim 1 , wherein the at least one camera is positioned on an automated carrier.

10. The system as claimed in claim 1 , wherein the at least one camera is positioned on a non-automated carrier.

11. The system as claimed in claim 1 , wherein the at least one camera is positioned on a bin.

12. The system as claimed in claim 1 , wherein the plurality of fiducials includes a first pair of light emitting diodes (LEDs) that are positioned on mutually opposing sides of the transverse portion and a second pair of LEDs that are positioned on mutually opposing sides of the transverse portion, wherein the first pair of LEDs and the second pair of LEDs are further positioned on mutually opposing sides of the end-effector, the first pair of LEDs being visible LEDs and the second pair of LEDs being infra-red LEDs, wherein the control circuitry included in the transverse portion of the calibration unit is configured to control the first pair of LEDs and the second pair of LEDs.

13. A method for providing extrinsic calibration of a camera to a relative working environment of a programmable motion device that includes an end-effector, said method comprising:
viewing with the camera a fiducial among a plurality of fiducials located on a calibration unit attached to the end-effector;
capturing a plurality of images containing the fiducial using the camera as the programmable motion device moves in at least three degrees of freedom; and
analyzing the plurality of images to determine a fiducial location with respect to the camera to permit calibration of the camera with the programmable motion device,
wherein the calibration unit includes a coupling portion for coupling to the end-effector and a transverse portion that is transverse to the coupling portion, wherein the plurality of fiducials are positioned on mutually opposing sides of the transverse portion of the calibration unit, the plurality of fiducials including a light emitting diode (LED) on a top side or a bottom side of the transverse portion, wherein the transverse portion further includes control circuitry for controlling the LED.

14. The method as claimed in claim 13 , wherein method includes providing intermittent visible illumination with the LED.

15. The method as claimed in claim 13 , wherein method includes providing intermittent infra-red illumination with the LED.

16. The method as claimed in claim 13 , wherein the plurality of fiducials further includes a retroreflective fiducial.

17. The method as claimed in claim 16 , wherein the retroreflective fiducial is at least partially spherically shaped.

18. The method as claimed in claim 13 , wherein analyzing the plurality of images includes constructing a matrix for each of the plurality of images of the fiducial.

19. The method as claimed in claim 18 , wherein analyzing the plurality of images includes concatenating extrinsic parameters of the camera to the programmable motion device.

20. The method as claimed in claim 19 , wherein analyzing the plurality of images includes employing non-linear least square optimization.

21. The method as claimed in claim 13 , wherein the camera is positioned on an automated carrier.

22. The method as claimed in claim 13 , wherein the camera is positioned on a non-automated carrier.

23. The method as claimed in claim 13 , wherein the camera is positioned on a bin.

24. The method as claimed in claim 13 , wherein viewing with the camera the fiducial as the programmable motion device moves in at least three degrees of freedom, capturing the plurality of images containing the fiducial, and analyzing the plurality of images to determine the fiducial location all occur during the processing of objects with the programmable motion device.

25. A method for providing extrinsic calibration of a camera to a relative working environment of a programmable motion device that includes an end-effector, said method comprising:
capturing a plurality of images containing a fiducial among a plurality of fiducials located on a calibration unit attached to the end-effector while the programmable motion device moves in at least three degrees of freedom; and
analyzing the plurality of images to determine a fiducial location with respect to the camera to permit calibration of the camera with the programmable motion device,
wherein the calibration unit including a coupling portion for coupling to the end-effector and a transverse portion that is transverse to the coupling portion, and
wherein the plurality of fiducials includes a first pair of light emitting diodes (LEDs) that are positioned on mutually opposing sides of the transverse portion of the calibration unit and a second pair of LEDs that are positioned on mutually opposing sides of the transverse portion, wherein the first pair of LEDs and the second pair of LEDs are further positioned on mutually opposing sides of the end-effector, the first pair of LEDs being visible LEDs and the second pair of LEDs being infra-red LEDs, wherein the transverse portion of the calibration unit further includes control circuitry for controlling the first pair of LEDs and the second pair of LEDs.

26. The method as claimed in claim 25 , wherein analyzing the plurality of images includes constructing a ma

Assignees

• Berkshire Grey Operating Company Inc

Inventors

• Ding Yu
• GEYER Christopher
• Galup Luis
• Kittredge Jeffrey
• Koletschka Thomas
• Kumar Lakshman
• Nellithimaru Anjana
• O'Hern Ryan
• Saslaw Jeremy

Classifications

• G05B2219/37009

  Calibration of vision system, camera, adapt light level · CPC title

• B25J9/1697 Primary

  Vision controlled systems · CPC title

• B25J9/1692

  Calibration of manipulator · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T7/80

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title