System and method for training a model in a plurality of non-perspective cameras and determining 3D pose of an object at runtime with the same

Patent metadata

Field	Value
Publication number	US-11488322-B2
Application number	US-96300710-A
Country	US
Kind code	B2
Filing date	Dec 8, 2010
Priority date	Dec 8, 2010
Publication date	Nov 1, 2022
Grant date	Nov 1, 2022

Abstract

This invention provides a system and method for training and performing runtime 3D pose determination of an object using a plurality of camera assemblies in a 3D vision system. The cameras are arranged at different orientations with respect to a scene, so as to acquire contemporaneous images of an object, both at training and runtime. Each of the camera assemblies includes a non-perspective lens that acquires a respective non-perspective image for use in the process. The searched object features in one of the acquired non-perspective image can be used to define the expected location of object features in the second (or subsequent) non-perspective images based upon an affine transform, which is computed based upon at least a subset of the intrinsics and extrinsics of each camera. The locations of features in the second, and subsequent, non-perspective images can be refined by searching within the expected location of those images. This approach can be used in training, to generate the training model, and in runtime operating on acquired images of runtime objects. The non-perspective cameras can employ telecentric lenses.

First claim

What is claimed is:

1. A method for determining a 3D pose of an object during runtime operation of a 3D vision system comprising the steps of:
orienting at least a first telecentric, non-perspective camera assembly and a second telecentric, non-perspective camera assembly with respect to the object to acquire a first non-perspective image by the first telecentric non-perspective camera assembly and acquire a second non-perspective image contemporaneously by the second telecentric non-perspective camera;
searching for 2D model features in the first non-perspective image using a first model;
searching for 2D model features in the second non-perspective image using a second model that is a descendant of the first model and is derived from an affine transform applied to the first model using 2D model features in the first non-perspective image acquired by the first telecentric non-perspective camera assembly, wherein the affine transform is based on at least a subset of the intrinsics and extrinsics associated with the first telecentric non-perspective camera assembly, the extrinsics comprising at least a pose of the first telecentric non-perspective camera assembly and the intrinsics comprising at least a pixel scale of the first telecentric non-perspective camera assembly; and
determining 3D pose of the object by determining:
(a) a mapping between 3D model points to locations of the 2D model features in the first non-perspective image that correspond to rays passing in parallel through a first telecentric non-perspective lens of the first telecentric non-perspective camera assembly, and
(b) a mapping between 3D model points to locations of the 2D model features in the second non-perspective image that correspond to rays passing in parallel through a second telecentric non-perspective lens of the second telecentric non-perspective camera assembly.

2. The method as set forth in claim 1 further comprising distorting at least one of the first non-perspective image and the second non-perspective image, according to a predetermined relationship, and removing lens distortion from at least one of the first non-perspective image and the second non-perspective image in order to perform the searching step respectively therein.

3. The method as set forth in claim 2 further comprising computing the affine transform based upon at least a subset of intrinsics and extrinsics of the second camera non-perspective camera assembly.

4. The method as set forth in claim 1 further comprising performing an operation on the object based upon the 3D pose.

5. The method as set forth in claim 4 wherein the operation includes manipulating a robot with respect to the object.

6. A 3D vision system for determining a 3D pose of an object during runtime operation comprising:
a first telecentric non-perspective camera assembly and a second telecentric non-perspective camera assembly that respectively acquire a first non-perspective image and a second non-perspective image of the object contemporaneously; and
a searching tool that searches for 2D model features in the first non-perspective image using a first model and that searches for 2D model features in the second non-perspective image using a second model that is a descendant of the first model and is derived from an affine transform applied to the first model using 2D model features in the first non-perspective image acquired by the first telecentric non-perspective camera assembly, wherein the affine transform is based on at least a subset of the intrinsics and extrinsics associated with the first non-perspective camera assembly, the extrinsics comprising at least a pose of the first telecentric non-perspective camera assembly and the intrinsics comprising at least a pixel scale of the first telecentric non-perspective camera, the 3D pose determined by:
(a) a mapping between 3D model points to locations of the searched 2D model features in each of the first non-perspective image that correspond to rays passing in parallel through a first telecentric non-perspective lens of the first telecentric non-perspective camera assembly, and
(b) a mapping between 3D model points to locations of the 2D model features in the second non-perspective image that correspond to rays passing in parallel through a second telecentric non-perspective lens of the second telecentric non-perspective camera assembly.

7. The 3D vision system as set forth in claim 6 wherein at least one of the first non-perspective image and the second non-perspective image is distorted according to a predetermined relationship upon acquisition, and further comprising an undistorting process that undistorts the at least one of the first non-perspective image and the second non-perspective image for searching by the searching tool.

8. The system as set forth in claim 6 wherein the affine transform is based upon at least a subset of intrinsics and extrinsics of the second camera non-perspective camera assembly.

9. The system as set forth in claim 6 further comprising a device that moves with respect to the object based upon the pose data.

10. The system as set forth in claim 9 wherein the device comprises a robot.

11. A method for training a model of an object for use during runtime operation of a 3D vision system comprising the steps of:
orienting at least a first telecentric non-perspective camera assembly and a second telecentric non-perspective camera assembly with respect to the object to acquire a first non-perspective image by the first telecentric non-perspective camera assembly and acquire a second non-perspective image contemporaneously by the second telecentric non-perspective camera assembly;
providing an affine transform between the first non-perspective camera assembly and the second telecentric non-perspective camera assembly by providing a first affine transform from an image plane of the first telecentric non-perspective camera assembly to an imaged plane and providing a second affine transform from the imaged plane to an image plane of the second telecentric non-perspective camera assembly using at least a subset of intrinsics and extrinsics of the first telecentric non-perspective camera assembly and intrinsics and extrinsics of the second telecentric non-perspective camera assembly;
defining a first model with a reference point in the first non-perspective image that correspond to rays passing through a first telecentric non-perspective lens of the first telecentric non-perspective camera assembly and locations in the second non-perspective image that correspond to rays passing in parallel through a second telecentric non-perspective lens of the second telecentric non-perspective camera assembly; and
deriving a second model as a descendant from the first model with the reference point by using the affine transform applied to the first model and at least a subset of the intrinsics and extrinsics associated with the first telecentric non-perspective camera assembly, the extrinsics comprising at least a pose of the first telecentric non-perspective camera assembly and the intrinsics comprising at least a pixel scale of the first telecentric non-perspective camera.

12. The method as set forth in claim 11 further comprising refining the second model based upon a search for the reference point in the second non-perspective image within a search range limited based upon the second model generated by the affine transform.

13. The method as set forth in claim 12 further comprising, during runtime, determining a 3D pose of the object based upon at least the first model and the second model.

14. The method as set forth in claim 13 further compri

Assignees

• Liu Lifeng
• Wallack Aaron S
• Marrion Jr Cyril C
• Cognex Corp

Inventors

• Liu Lifeng
• Wallack Aaron S
• Marrion Jr Cyril C

Classifications

• G06T7/75 Primary

  involving models · CPC title