Determining soil state and controlling equipment based on captured images

Patent metadata

Field	Value
Publication number	US-10867377-B2
Application number	US-201715851526-A
Country	US
Kind code	B2
Filing date	Dec 21, 2017
Priority date	Dec 21, 2016
Publication date	Dec 15, 2020
Grant date	Dec 15, 2020

Abstract

Methods and systems for controlling earth moving equipment are described. In some embodiments, a stream of images may be captured using at least one imaging system. At least one state of soil may be determined in real time relative to the earth moving equipment based on the captured stream of images. At least one aspect of the earth moving equipment's operation may then be controlled based on the determined at least one state of the soil.

First claim

What is claimed is:

1. A piece of earth moving equipment, the earth moving equipment comprising:
at least one container; and
at least one processor configured to:
determine at least one state of soil, including at least one flow rate of the soil relative to the at least one container of the earth moving equipment, in real time relative to the earth moving equipment based on a captured stream of images at least by evaluating differential motion between two or more image frames of the captured stream of images,
wherein evaluating the differential motion comprises identifying one or more features within the two or more image frames and determining a relative motion of at least one of the one or more features; and
control, without need of a human operator, at least one aspect of the earth moving equipment's operation based on the determined at least one state of the soil at least by selecting between digging and scooping of the soil with the at least one container of the earth moving equipment based on recognizing a pattern in soil movement using at least one neural network.

2. The earth moving equipment of claim 1 , further comprising at least one imaging system, and wherein the at least one processor is configured to capture the stream of images from the at least one imaging system.

3. The earth moving equipment of claim 2 , wherein the at least one processor is configured to:
determine the at least one state of the soil at least by determining a total amount of soil in the at least one container of the earth moving equipment based on the at least one flow rate; and
determine a fill rate of the at least one container of the earth moving equipment based on the at least one flow rate.

4. The earth moving equipment of claim 1 , wherein the at least one state of the soil comprises at least one property of the soil including at least one of a shear failure surface of the soil, a friction angle of the soil, and a granularity of the soil.

5. The earth moving equipment of claim 1 , wherein the at least one processor is configured to:
detect areas of soil movement within a field of view; and
determine the at least one state of the soil based on the detected areas of soil movement within the field of view.

6. The earth moving equipment of claim 1 , wherein the at least one processor is further configured to:
detect soil movement in front of a leading edge of the at least one container of the earth moving equipment; and
determine the at least one state of the soil based on the detected soil movement in front of the leading edge of the at least one container of the earth moving equipment.

7. The earth moving equipment of claim 1 , wherein the at least one processor is further configured to update a flow-to-volume model via learning based on a measurement of an actual total amount of soil in the at least one container of the earth moving equipment.

8. A method for controlling earth moving equipment, the method comprising:
determining at least one state of soil, including at least one flow rate of the soil relative to at least one container of the earth moving equipment, in real time relative to the earth moving equipment based on a captured stream of images at least by evaluating differential motion between two or more image frames of the captured stream of images, wherein evaluating the differential motion comprises identifying one or more features within the two or more image frames and determining a relative motion of at least one of the one or more features; and
controlling, without need of a human operator, at least one aspect of the earth moving equipment's operation based on the determined at least one state of the soil, at least by selecting between digging and scooping of the soil with the at least one container of the earth moving equipment based on recognizing a pattern in soil movement using at least one neural network.

9. The method of claim 8 , further comprising capturing the stream of images from at least one imaging system.

10. The method of claim 8 , further comprising:
detecting areas of soil movement within a field of view; and
determining the at least one state of the soil based on the detected areas of soil movement within the field of view.

11. The method of claim 8 , further comprising updating a flow-to-volume model via learning based on a measurement of an actual total amount of soil in the at least one container of the earth moving equipment.

12. At least one non-transitory computer-readable medium having instructions thereon that, when executed by at least one processor, perform a method for controlling earth moving equipment, the method comprising:
determining at least one state of soil, including at least one flow rate of the soil relative to at least one container of the earth moving equipment, in real time relative to the earth moving equipment based on a captured stream of images at least by evaluating differential motion between two or more image frames of the captured stream of images, wherein evaluating the differential motion comprises identifying one or more features within the two or more image frames and determining a relative motion of at least one of the one or more features; and
controlling, without need of a human operator, at least one aspect of the earth moving equipment's operation based on the determined at least one state of the soil, at least by selecting between digging and scooping of the soil with the at least one container of the earth moving equipment based on recognizing a pattern in soil movement using at least one neural network.

13. The at least one non-transitory computer-readable medium of claim 12 , the method further comprising capturing a stream of images from at least one imaging system.

14. The at least one non-transitory computer-readable medium of claim 12 , wherein determining the at least one state of the soil comprises determining a total amount of soil in the at least one container of the earth moving equipment based on at least one flow rate, and the method comprises determining a fill rate of the at least one container of the earth moving equipment based on the at least one flow rate.

15. The at least one non-transitory computer-readable medium of claim 12 , the method further comprising:
detecting areas of soil movement within a field of view; and
determining the at least one state of the soil based on the detected areas of soil movement within the field of view.

16. The at least one non-transitory computer-readable medium of claim 12 , the method further comprising:
detecting soil movement in front of a leading edge of the at least one container of the earth moving equipment; and
determining the at least one state of the soil based on the detected soil movement in front of the leading edge of the at least one container of the earth moving equipment.

17. The at least one non-transitory computer-readable medium of claim 12 , the method further comprising updating a flow-to-volume model via learning based on a measurement of an actual total amount of soil in the at least one container of the earth moving equipment.

18. The earth moving equipment of claim 1 , wherein evaluating the differential motion comprises applying phase correlation, a two-dimensional median filter, and/or flow adapted anisotropic diffusion.

19. The method of claim 8 , wherein evaluating the differential motion comprises applying phase correlation, a two-dimensional median filter, and/or flow adapted anisotropic diffusion.

20. The at least one non-transitory computer-

Assignees

• Massachusetts Inst Technology

Inventors

• Asada Haruhiko Harry
• Sotiropoulos Filippos Edward

Classifications

• G01N33/24 Primary

  Earth materials (G01N33/42 takes precedence) · CPC title

• E02F9/261

  Surveying the work-site to be treated · CPC title

• G06T7/0004

  Industrial image inspection · CPC title

• G06T7/97

  Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85) · CPC title

• G06T7/246

  using feature-based methods, e.g. the tracking of corners or segments · CPC title