Camera array calibration in a farming machine

Patent metadata

Field	Value
Publication number	US-11961259-B2
Application number	US-202117566396-A
Country	US
Kind code	B2
Filing date	Dec 30, 2021
Priority date	Dec 30, 2021
Publication date	Apr 16, 2024
Grant date	Apr 16, 2024

Abstract

The calibration system of the farming machine receives images from each camera of the camera array. The images comprise visual information representing a view of a portion of an area surrounding the farming machine. To calibrate a pair of cameras including a first camera and second camera, the calibration system determines a relative pose between the pair of cameras by extracting relative position and orientation characteristics from visual information in both an image received from the first camera and an image received from the second camera. The calibration system identifies a calibration error for the pair of cameras based on a comparison of the relative pose with an expected pose between the first pair of cameras. The calibration system transmits a notification to an operator of the farming machine that describes the calibration error and instructions for remedying the calibration error.

First claim

What is claimed is:

1. A method for validating a calibration of a plurality of cameras on a farming machine having a mounting mechanism, the method comprising:
for each camera of the plurality of cameras on the farming machine, receiving an image comprising visual information representing a view of a portion of an area surrounding the farming machine;
for a first pair of cameras comprising a first camera of the plurality of cameras and a second camera of the plurality of cameras,
determining a first relative pose between the first pair of cameras, wherein the first relative pose comprises a position and an orientation of the first camera relative to the second camera by extracting relative position and orientation characteristics from information in both a first image received from the first camera and a second image received from the second camera;
identifying a calibration error for the first pair of cameras based on a comparison of the first relative pose with an expected relative pose between the first pair of cameras, wherein the expected relative pose is described by a virtual representation of the farming machine and comprises an expected position and an expected orientation of the first camera relative to the second camera; and
transmitting, to an operator of the farming machine, a notification comprising the calibration error and instructions to remedy the calibration error.

2. The method of claim 1 , further comprising:
accessing the first relative pose and a second relative pose between a second pair of cameras, the second pair of cameras comprising the second camera and a third camera of the plurality of cameras;
determining that the second camera is adjacent to both the first camera and the third camera based on a comparison of the first relative pose and the second relative pose; and
responsive to determining that the second camera is adjacent to both the first camera and the third camera, updating the calibration error based on the second relative pose.

3. The method of claim 1 , wherein visual information of the first image comprises one or more parts of the farming machine, and the method further comprises:
determining a first position of the first camera on the farming machine based on visual information of the first image representing the one or more parts of the farming machine;
determining a second position of the second camera on the farming machine based on the position of the first camera; and
determining the first relative pose based on the determined first position and the determined second position.

4. The method of claim 1 , further comprising:
identifying, from the plurality of cameras, a first end camera positioned at a first end of the mounting mechanism and a second end camera positioned at a second end of the mounting mechanism based on hardware addresses defined in the virtual representation of the farming machine for each camera of the plurality of cameras; and
calibrating the first end camera and the second end camera by:
determining a second calibration error between the first end camera and the second end camera based on a comparison of a second relative pose between the first end camera and the second end camera with an expected relative pose between the first end camera and the second end camera; and
distributing the calibration error across a plurality of camera pairs of the plurality of cameras located between the first end camera and the second end camera.

5. The method of claim 1 , wherein the first image and the second image comprise a part of the farming machine, and the method further comprises:
extracting position characteristics from visual information of the first image representing the part of the farming machine, the extracted position characteristics describing the pose of the first camera relative to the part of the farming machine;
extracting position characteristics from visual information of the second image representing the part of the farming machine, the extracted position characteristics describing the pose of the second camera relative to the part of the farming machine; and
determining the first relative pose by comparing the extracted position characteristics from the visual information of the first image and the visual information of the second image.

6. The method of claim 1 , wherein the first image and the second image comprise a fiducial marker in the portion of the area surrounding the farming machine, and the method further comprises:
extracting position characteristics from visual information of the first image representing the fiducial marker, the extracted position characteristics describing the pose of the first camera relative to the fiducial marker;
extracting position characteristics from visual information of the second image representing the fiducial marker, the extracted position characteristics describing the pose of the second camera relative to the fiducial marker; and
determining the first relative pose by comparing the extracted position characteristics from the visual information of the first image to the extracted position characteristics of the visual information of the second image.

7. The method of claim 1 , further comprising:
accessing, from the virtual representation of the farming machine, a first position associated with a first processor operating the first camera using on a first hardware address of the first camera;
accessing, from the virtual representation of the farming machine, a second position associated with a second processor operating the second camera using on a second hardware address of the second camera; and
determining the expected relative pose between the first pair of cameras by comparing first position associated with the first hardware address and the second position associated with the second hardware address.

8. The method of claim 1 , wherein determining the first relative pose comprises:
measuring, by a sensor mounted on the farming machine, a first height of the first camera relative to a ground surface and a second height of the second camera relative to the ground surface;
determining a scale factor for relative pose determination based on the measured first height and second height; and
determining the first relative pose based on the scale factor.

9. The method of claim 1 , wherein determining the first relative pose comprises:
determining a first angle of the first camera relative to a ground surface and a second angle of the second camera relative to the ground surface; and
determining the first relative pose based on the first angle and the second angle.

10. The method of claim 1 , wherein determining the first relative pose comprises:
determining a scale factor for the first camera and the second camera representing a quantification of a relationship between distances captured in the first image and the second image and actual distances in the area surrounding the farming machine; and
determining the relative first pose between the first pair of cameras based on the scale factor.

11. The method of claim 1 , wherein the second camera is positioned on the farming machine adjacent to the first camera, and the method further comprises:
comparing visual information of the first image to visual information of the second image;
responsive to determining the visual information of the first image does not overlap with the visual information of the second image, comparing the visual information of the first image to visual information of images captured by other cameras of the plurality of cameras;
identifying, from the images captured by the other cameras, a third camera that captured an image with visual information overlapping wit

Assignees

• Blue River Tech Inc

Inventors

• Ostrowski James Patrick
• Peake John William
• Song Jacqueline Gao

Classifications

• G06T7/85 Primary

  Stereo camera calibration · CPC title

• A01B69/001

  Steering by means of optical assistance, e.g. television cameras (steering devices for road marking vehicles E01C23/163) · CPC title

• G06T7/73

  using feature-based methods · CPC title

• G06V20/188

  Vegetation · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title