Method and apparatus for controlling excavator to excavate

Patent metadata

Field	Value
Publication number	US-11668076-B2
Application number	US-201916677308-A
Country	US
Kind code	B2
Filing date	Nov 7, 2019
Priority date	Jan 25, 2019
Publication date	Jun 6, 2023
Grant date	Jun 6, 2023

Abstract

Embodiments of the present disclosure relate to a method and apparatus for controlling an excavator to excavate. The method includes: acquiring a two-dimensional image of a material pile; generating a three-dimensional model of the material pile based on the two-dimensional image; analyzing the three-dimensional model to determine a target excavating point and a target excavating trajectory of the material pile; and controlling an excavator to excavate a material at the target excavating point along the target excavating trajectory.

First claim

What is claimed is:

1. A method for controlling an excavator to excavate, comprising:
acquiring a two-dimensional image of a material pile;
generating a three-dimensional model of the material pile based on the two-dimensional image;
analyzing the three-dimensional model to determine a target excavating point on the material pile and a target excavating trajectory of a bucket of the excavator, wherein the target excavating trajectory is decomposed into four parts: insertion, dragging, rotation, and lifting, and the target excavating trajectory is characterized by four parameters: an insertion depth of the bucket in the material pile, a dragging length of the bucket in the material pile, a rotation radius of the bucket, and a lifting height of the bucket; and
controlling the excavator to excavate a material at the target excavating point along the target excavating trajectory, comprising:
determining a target angle sequence for a base and an excavating arm of the excavator to excavate the material at the target excavating point along the target excavating trajectory, wherein the target angle sequence includes a sequence of target angles of the base and the excavation arm at sampling points on the target excavating trajectory; and
controlling rotation angles and displacements of the base and the excavating arm based on the target angle sequence.

2. The method according to claim 1 , wherein a side of a body of the excavator is provided with a camera, and the camera captures the two-dimensional image when the body is rotated to unload the material in the bucket.

3. The method according to claim 2 , wherein the camera is a binocular camera, and the two-dimensional image is a binocular image; and
the generating a three-dimensional model of the material pile based on the two-dimensional image comprises:
performing image matching on the binocular image to obtain a parallax image of the binocular image;
generating a depth image of the material pile based on the parallax image; and
performing three-dimensional reconstruction on the depth image to obtain the three-dimensional model of the material pile.

4. The method according to claim 1 , wherein the analyzing the three-dimensional model to determine a target excavating point and a target excavating trajectory of the material pile comprises:
discretizing the three-dimensional model to obtain an excavating point set;
acquiring an excavating trajectory set of an excavating point in the excavating point set;
selecting a candidate excavating point set based on the excavating trajectory set of the excavating point in the excavating point set;
acquiring a material volume of a candidate excavating point in the candidate excavating point set; and
selecting a target excavating point and a target excavating trajectory based on the material volume of the candidate excavating point in the candidate excavating point set.

5. The method according to claim 1 , wherein the controlling an excavator to excavate a material at the target excavating point along the target excavating trajectory further comprises:
acquiring current angles of the base and the excavating arm from a base angle sensor and an excavating arm sensor of the excavator in real time, and performing closed-loop control based on a size and a positive or negative characteristic of a difference between a current target angle in the target angle sequence and the current angles, so that the base and the excavating arm excavate the material at the target excavating point along the target excavating trajectory.

6. An apparatus for controlling an excavator to excavate, comprising:
at least one processor; and
a memory storing instructions, the instructions when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising:
acquiring a two-dimensional image of a material pile;
generating a three-dimensional model of the material pile based on the two-dimensional image;
analyzing the three-dimensional model to determine a target excavating point on the material pile and a target excavating trajectory of a bucket of the excavator, wherein the target excavating trajectory is decomposed into four parts: insertion, dragging, rotation, and lifting, and the target excavating trajectory is characterized by four parameters: an insertion depth of the bucket in the material pile, a dragging length of the bucket in the material pile, a rotation radius of the bucket, and a lifting height of the bucket; and
controlling the excavator to excavate a material at the target excavating point along the target excavating trajectory, comprising:
determining a target angle sequence for a base and an excavating arm of the excavator to excavate the material at the target excavating point along the target excavating trajectory, wherein the target angle sequence include a sequence of target angles of the base and the excavation arm at sampling points on the target excavating trajectory; and
controlling rotation angles and displacements of the base and the excavating arm based on the target angle sequence.

7. The apparatus according to claim 6 , wherein a side of a body of the excavator is provided with a camera, and the camera captures the two-dimensional image when the body is rotated to unload the material in the bucket.

8. The apparatus according to claim 7 , wherein the camera is a binocular camera, and the two-dimensional image is a binocular image; and
the generating a three-dimensional model of the material pile based on the two-dimensional image comprises:
performing image matching on the binocular image to obtain a parallax image of the binocular image;
generating a depth image of the material pile based on the parallax image; and
performing three-dimensional reconstruction on the depth image to obtain the three-dimensional model of the material pile.

9. The apparatus according to claim 6 , wherein the analyzing the three-dimensional model to determine a target excavating point and a target excavating trajectory of the material pile comprises:
discretizing the three-dimensional model to obtain an excavating point set;
acquiring an excavating trajectory set of an excavating point in the excavating point set;
selecting a candidate excavating point set based on the excavating trajectory set of the excavating point in the excavating point set;
acquiring a material volume of a candidate excavating point in the candidate excavating point set; and
selecting a target excavating point and a target excavating trajectory based on the material volume of the candidate excavating point in the candidate excavating point set.

10. The apparatus according to claim 6 , wherein the controlling an excavator to excavate a material at the target excavating point along the target excavating trajectory further comprises:
acquiring current angles of the base and the excavating arm from a base angle sensor and an excavating arm sensor of the excavator in real time, and performing closed-loop control based on a size and a positive or negative characteristic of a difference between a current target angle in the target angle sequence and the current angles, so that the base and the excavating arm excavate the material at the target excavating point along the target excavating trajectory.

11. A non-transitory computer readable medium, storing a computer program, wherein the computer program, when executed by a processor, causes the processor to perform operations for controlling an excavator to excavate, the operations comprising:
acquiring a two-dimensional image of a material pile;
generating a three-dimensional model of the material pile based on the two-dim

Assignees

• Beijing Baidu Netcom Sci & Tech Co Ltd

Inventors

• Cheng Xinjing
• Yang Ruigang
• Yang Yajue
• Lu Feixiang
• Xu Hao

Classifications

• E02F3/435 Primary

  for dipper-arms, backhoes or the like · CPC title

• E02F9/205

  Remotely operated machines, e.g. unmanned vehicles (E02F3/8866 takes precedence) · CPC title

• E02F9/26

  Indicating devices {(E02F5/145 takes precedence)} · CPC title

• E02F9/2054

  Fleet management · CPC title

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title