Mobile work machine with object detection and machine path visualization

Patent metadata

Field	Value
Publication number	US-11814816-B2
Application number	US-201916567789-A
Country	US
Kind code	B2
Filing date	Sep 11, 2019
Priority date	Sep 11, 2019
Publication date	Nov 14, 2023
Grant date	Nov 14, 2023

Abstract

A method of controlling a mobile work machine on a worksite including receiving an indication of an object detected on the worksite, determining a location of the object relative to the mobile work machine, receiving an image of the worksite, correlating the determined location of the object to a portion of the image, and generating a control signal that controls a display device to display a representation of the image with a visual object indicator that represents the detected object on the portion of the image.

First claim

What is claimed is:

1. A method of controlling a mobile work machine on a worksite, the method comprising:
receiving an indication of a detected object detected on the worksite;
determining a location of the detected object relative to the mobile work machine;
receiving an image of the worksite to a rear of the mobile work machine from an imaging sensor on the mobile work machine;
identifying a portion of the image that is correlated to the location of the detected object;
receiving a control input to control the mobile work machine to traverse the worksite in a rearward direction;
determining a projected rearward path of the mobile work machine based on the control input;
setting a visual characteristic for a visual path indicator based on a determination that at least a portion of the detected object is located in the projected rearward path; and
generating a control signal that controls a display device to display on the display device:
a representation of the image,
the visual path indicator, having the visual characteristic, that represents the projected rearward path on the image, and
a visual object indicator that represents the detected object and is displayed on the portion of the image.

2. The method of claim 1 , and further comprising:
performing the image processing to determine a window size for the window within which a closed blob is detected; and
identifying the detected object based on detection of the closed blob.

3. The method of claim 2 , and further comprising:
performing blob segmentation that separates the detected object from an image background; and
iteratively increasing the window size until the closed blob is detected.

4. The method of claim 1 , wherein setting a visual characteristic of the visual path indicator comprises selecting a display color of the visual path indicator.

5. The method of claim 1 , and further comprising:
changing the visual path indicator to a first color in response to determining that the detected object is not located in the projected rearward path; and
changing the visual path indicator to a second color, different from the first color, in response to determining that the detected object is located in the projected rearward path.

6. The method of claim 1 , wherein the mobile work machine comprises ground engaging traction elements, and further comprising:
determining a differential drive speed of a first ground engaging traction element relative to a second ground engaging traction element; and
determining the projected rearward path of the mobile wok machine based on the differential drive speed.

7. The method of claim 1 , wherein the mobile work machine comprises an articulated machine, and further comprising:
determining the projected rearward path of the mobile work machine based on an articulation angle of the articulated machine.

8. The method of claim 1 , wherein the indication of the detected object is received from a non-optical object detection Sensor on the mobile work machine.

9. The method of claim 8 , wherein the non-optical object detection sensor comprises a radio detection and ranging (RADAR) sensor.

10. The method of claim 1 , and further comprising:
evaluating the detected object by performing image processing of the portion of the image; and
generating the visual object indicator based on the evaluation.

11. The method of claim 8 , and further comprising:
evaluating the detected object by determining a likelihood that detection of the detected object by the non-optical object detection sensor comprises a false positive detection; and
the visual object indicator comprises a false positive indicator based on the likelihood.

12. The method of claim 1 , and further comprising:
applying an image classifier to the portion of the image to determine an object type of the detected object; and
displaying the visual object indicator based on the object type.

13. The method of claim 1 , wherein
the mobile work machine comprises a frame, an operator compartment configured to orient an operator in a forward direction of travel of the mobile work machine, and a tool supported by the frame and configured to move material on the worksite, and
the imaging sensor comprises a rear-facing camera on the mobile work machine, the rear-facing camera being oriented in a rearward direction that is opposite the forward direction.

14. A mobile work machine comprising:
an object detection sensor configured to:
transmit a detection signal;
receive a reflection of the detection signal; and
based on the reflection, generate a first sensor signal indicative of a relative position of a detected object to a sensor location of the object detection sensor;

an optical imaging sensor configured to generate a second sensor signal representing an image of a worksite to a rear of the mobile work machine;
at least one processor; and
memory storing instructions executable by the at least one processor, wherein the instructions, when executed, cause the mobile work machine to:
correlate the relative position of the detected object to a portion of the image;
generate object characterization data representing the detected object based on image processing on the portion of the image;
determine a projected rearward path of the mobile work machine; and
generate a control signal that controls a display device to:
display a representation of the image with
a visual path indicator indicative of the projected rearward path on the image; and
a visual object indicator that represents the detected object on the portion of the image based on the object characterization data, and

change a visual characteristic of the visual path indicator based on a determination that at least a portion of the detected object is located in the projected rearward path.

15. The mobile work machine of claim 14 , wherein the instructions, when executed, cause the mobile work machine to:
perform the image processing to determine a window size for the window within which a closed blob is detected and
identify the detected object based on detection of the closed blob.

16. The mobile work machine of claim 14 , wherein the mobile work machine comprises ground engaging Unction elements, and the instructions cause the mobile work machine to determine the projected rearward path based on at least one of:
a differential drive speed of a first ground engaging traction element relative to a second ground engaging traction element; or
an articulation angle of the mobile work machine.

17. A mobile work machine comprising:
a set of ground engaging traction elements configured to move the mobile work machine over a worksite a forward direction of travel;
an operator compartment having a display device;
an object detection sensor configured to:
a transmitter configured to transmit an electromagnetic signal; and
a receiver configured to receive a reflection of the electromagnetic signal and to generate, based on the reflection, an object indication signal;

a rear-facing camera having a field of view in a rearward direction that is opposite the forward direction, the rear-facing camera configured to capture images with the field of view;
at least one processor; and
memory storing instructions executable by the at least one processor, wherein the instructions, when executed, cause the mobile work machine to:
based on the object indication signal from the object detection sensor, identify a window within the field of view of the rear-facing camera;
perform image processing on the window;
identify a de

Assignees

• Deere & Co

Inventors

• Kean Michael G
• Harwell Drew F
• Darling Cale

Classifications

• E02F9/265 Primary

  with follow-up actions (e.g. control signals sent to actuate the work tool) · CPC title

• B60R1/00

  Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles · CPC title

• E02F9/262

  with follow-up actions to control the work tool, e.g. controller · CPC title

• G05D1/0094

  involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target · CPC title

• G05D1/0219

  ensuring the processing of the whole working surface · CPC title