Container angle sensing using vision sensor for feedback loop control

Patent metadata

Field	Value
Publication number	US-10689830-B2
Application number	US-201816055495-A
Country	US
Kind code	B2
Filing date	Aug 6, 2018
Priority date	Aug 6, 2018
Publication date	Jun 23, 2020
Grant date	Jun 23, 2020

Abstract

A mobile work machine includes a frame and a ground engaging element movably supported by the frame and driven by an engine to drive movement of the mobile work machine. The mobile work machine also includes a container movably supported by the frame and an actuator configured to controllably drive movement of the container relative to the frame. The work machine also includes a control system configured to generate an actuator control signal, indicative of a commanded movement of the actuator, and provide the actuator control signal to the actuator to control the actuator to perform the commanded movement and an image sensor coupled to the mobile work machine, the image sensor being configured to capture an image of the container. The work machine also includes an angle determination system, communicatively coupled to the control system, configured to determine an angle of the container relative to the image sensor, based on the image of the container.

First claim

What is claimed is:

1. A mobile work machine comprising:
a frame;
a ground engaging element movably supported by the frame and driven by an engine to drive movement of the mobile work machine;
a container movably supported by the frame;
an actuator configured to controllably drive movement of the container relative to the frame;
a control system configured to generate an actuator control signal, indicative of a commanded movement of the actuator, and provide the actuator control signal to the actuator to control the actuator to perform the commanded movement;
an image sensor coupled to the mobile work machine, the image sensor being configured to capture an image of the container;
segmentation logic configured to identify a segment of the image that includes the container;
area generator logic configured to calculate an area of the segment; and
an angle determination system, communicatively coupled to the control system, configured to determine an angle of the container, based on the area of the segment.

2. The mobile work machine of claim 1 , wherein the segmentation logic generates a set of container perimeter coordinates based on the segment; and wherein the angle determination system comprises:
profiling logic configured to confirm the angle from angle generator logic by comparing the set of container perimeter coordinates with a set of predetermined reference coordinates.

3. The mobile work machine of claim 2 , wherein the angle determination system comprises:
reference generator logic configured to generate the set of predetermined reference coordinates by receiving a plurality of different reference images from the image sensor, and receiving a plurality of different reference angles, each different reference image having a corresponding different reference angle indicative of an angle of the container in the difference reference image.

4. The mobile work machine of claim 1 , wherein the image sensor comprises at least one of a stereo camera or a laser scanner.

5. The mobile work machine of claim 1 , further comprising:
a volume generator system, coupled to the control system, configured to determine a volume of contents in the container, based on the angle of the container and the image.

6. The mobile work machine of claim 5 , further comprising:
display generator logic configured to display the volume of the contents in the container on a display device in a cab of the mobile work machine.

7. The mobile work machine of claim 5 , wherein the container comprises a bucket, the mobile work machine comprises an excavator, and the contents comprise earth.

8. A method of controlling a mobile work machine, comprising:
receiving, with a control system, an operator input indicative of a commanded movement of an actuator configured to drive movement of a container movably supported by a frame of the mobile work machine;
generating, with the control system, a control signal indicative of the commanded movement;
receiving, with angle generator logic and from a 3D sensor, an image of the container of the mobile work machine; and
determining, with the angle generator logic, an angle of the container with respect to the 3D sensor, wherein determining the angle of the container comprises:
identifying and segmenting a portion the image, the portion of the image comprising the container;
determining an area of the portion with area generator logic;
converting the area to the angle, based on a pre-determined correlation between the area and angle.

9. The method of claim 8 , further comprising:
verifying the angle of the container, with angle verification logic, by comparing the image against a set of reference images.

10. The method of claim 8 , wherein segmenting the portion of the image comprises, generating a convex hull.

11. A mobile work machine control system, comprising:
control logic configured to generate an actuator control signal, indicative of a commanded movement of an actuator coupled to a container of the mobile work machine to controllably drive movement of the container of the mobile work machine, and provide the actuator control signal to the actuator to control the actuator to perform the commanded movement;
a sensor configured to capture a captured image of contents in the container of the mobile work machine;
reference generator logic configured to receive a reference image of an empty container;
segmentation logic configured to identify a segment of the captured image comprising the container, wherein the segmentation logic identifies the container in the captured image, based on the reference image;
area generator logic configured to determine an area of the segment in the captured image; and
angle generator logic, coupled to the control logic, configured to determine an angle of the container with respect to the sensor, based on the area; and
angle verification logic configured to verify the angle determined by angle generator logic by comparing the captured image to the reference image, wherein a reference container angle corresponding to the reference image is known.

12. The mobile work machine control system of claim 11 , further comprising:
volume generator logic, coupled to the control logic, configured to determine a volume of contents in the container, based on the angle and captured image.

Assignees

• Deere & Co

Inventors

• Hageman John M
• Loukili Tarik
• Amatya Suraj

Classifications

• G01F22/00

  Methods or apparatus for measuring volume of fluids or fluent solid material, not otherwise provided for · CPC title

• E02F3/6454

  Towed (i.e. pulled or pushed) scrapers · CPC title

• G01C9/06

  Electric or photoelectric indication or reading means · CPC title

• H04N13/239

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title

• E02F9/264 Primary

  Sensors and their calibration for indicating the position of the work tool · CPC title