3-D image system for vehicle control

Patent metadata

Field	Value
Publication number	US-12554262-B2
Application number	US-202418620744-A
Country	US
Kind code	B2
Filing date	Mar 28, 2024
Priority date	Jun 22, 2017
Publication date	Feb 17, 2026
Grant date	Feb 17, 2026

Abstract

A control system uses visual odometry (VO) data to identify a position of the vehicle while moving along a path next to the row and to detect the vehicle reaching an end of the row. The control system can also use the VO image to turn the vehicle around from a first position at the end of the row to a second position at a start of another row. The control system may detect an end of row based on 3-D image data, VO data, and GNSS data. The control system also may adjust the VO data so the end of row detected from the VO data corresponds with the end of row location identified with the GNSS data.

First claim

The invention claimed is:

1 . A control system for controlling a machine having 1) a first sensor comprising a camera, 2) a second sensor that is different than the first sensor, and 3) an auto-steering system, the control system comprising:
one or more hardware processors configured to:
obtain map information indicative of a desired predefined path generated by a path generator, the desired predefined path extending along rows in a field;
receive image data for a row of the rows and identify two maps, including:
a localization map usable by the machine when the machine is able to localize itself within the row; and
an additional map usable by the machine when the machine is not able to localize itself within the row;

generate visual odometry (VO) data from the image data, wherein the VO data comprises non-localized data;
use the non-localized data or the additional map to identify a position of the machine while moving along a physical path next to the row; and
use the non-localized data or the additional map to detect the machine reaching an end of the row, and if sensor data is collected by the second sensor, localize VO calculations of the non-localized data in response to collection of the sensor data; and
wherein the localized VO calculations comprise localized data, and the one or more hardware processors are further configured to:
use the localized data, if available, or otherwise use the non-localized data, to turn the machine around from a first position at the end of the row to a second position at a start of another row of the rows.

2 . The control system of claim 1 , wherein the one or more hardware processors are further configured to:
detect an orientation of the machine at the end of the row based on the localized data, if available, or otherwise based on the non-localized data; and plan the turn of the machine based on the detected orientation of the machine.

3 . The control system of claim 1 , wherein the one or more hardware processors are further configured to:
monitor the image data to detect the end of the row; monitor the VO data to detect the end of the row; monitor global navigation satellite system (GNSS) data received from a GNSS receiver to detect the end of the row, wherein the second sensor comprises the GNSS receiver; and detect the end of the row based on any combination of the image data, the VO data, and the GNSS data.

4 . The control system of claim 3 , wherein the one or more hardware processors are further configured to:
determine a first probability for a first end of row location identified from the image data; determine a second probability for a second end of row location identified from the VO data; determine a third probability for a third end of row location identified from the GNSS data; and determine the end of the row based on the first, second, and third probabilities.

5 . The control system of claim 4 , wherein the one or more hardware processors are further configured to disregard the first end of row location when the first probability for the first end of row location does not overlap the second or third probabilities and is below a predetermined threshold.

6 . The control system of claim 4 , wherein the one or more hardware processors are further configured to identify the end of the row when any combination of the first, second, and third probabilities exceed a predetermined threshold.

7 . The control system of claim 4 , wherein the one or more hardware processors are further configured to:
identify one of the first, second, and third end of row locations preceding or exceeding another one of the first, second, and third end of row locations by a predetermined amount; and send a notification identifying the preceding or exceeding one of the first, second, and third end of row locations.

8 . The control system of claim 1 , wherein the one or more hardware processors are further configured to:
receive global navigation satellite system (GNSS) data from a GNSS receiver identifying an end of row location, wherein the second sensor comprises the GNSS receiver; and adjust the VO data so the end of the row corresponds with the end of row location identified with the GNSS data.

9 . The control system of claim 1 , wherein the one or more hardware processors are further configured to:
generate a point cloud map from the image data, the point cloud map identifying two adjacent rows in the field; generate two lines corresponding with locations of the two adjacent rows; generate a centerline between the two lines, the centerline indicating a desired path for the machine between the two adjacent rows; and detect the machine reaching the end of the row based on the VO data and the point cloud map.

10 . The control system of claim 1 , wherein the one or more hardware processors are further configured to adjust the VO data so:
1) an end of row location from the VO data corresponds with an end of row location from the sensor data, and 2) one or more additional locations, between the rows, are adjusted based at least in part on a difference between the end of row locations; and wherein the one or more hardware processors are further configured control the auto-steering system using the adjusted VO data to turn the machine around from the first position at the end of the row to the second position at the start of the other row.

Assignees

• Agjunction Llc

Inventors

• Madsen Tommy Ertboelle
• Sakharkar Anant

Classifications

• G05D1/2435

  Extracting 3D information · CPC title

• G05D1/243

  Means capturing signals occurring naturally from the environment, e.g. ambient optical, acoustic, gravitational or magnetic signals (using passive navigation aids external to the vehicle G05D1/244; using signals from positioning sensors located off-board the vehicle G05D1/249) · CPC title

• G05D1/227

  Handing over between remote control and on-board control; Handing over between remote control arrangements · CPC title

• G05D1/24

  Arrangements for determining position or orientation · CPC title

• G01C21/30

  Map- or contour-matching · CPC title