Three-dimensional reconstruction method and apparatus for material pile, electronic device, and computer-readable medium

Patent metadata

Field	Value
Publication number	US-11568593-B2
Application number	US-201916676281-A
Country	US
Kind code	B2
Filing date	Nov 6, 2019
Priority date	Jan 25, 2019
Publication date	Jan 31, 2023
Grant date	Jan 31, 2023

Abstract

Embodiments of the present disclosure relate to a three-dimensional reconstruction method and apparatus for a material pile, an electronic device, and a computer-readable medium. The method may include: acquiring, in response to an instruction for controlling an excavator body of an excavator to rotate to transport materials being detected, a sequence of depth images of an excavated material pile collected by a binocular camera provided on a side of the excavator; and performing three-dimensional reconstruction based on the sequence of depth images of the material pile, to generate a three-dimensional model of the material pile.

First claim

What is claimed is:

1. A three-dimensional reconstruction method for a material pile, applied to an unmanned autonomous excavator, where a binocular camera is provided on each of two side-surfaces of the excavator, and the two side-surfaces of the excavator are parallel to an advancing direction of the excavator, the method comprising:
in response to detecting an instruction for controlling an excavator body of the excavator to rotate to transport materials shoveled from the material pile, determining a side-surface turned to the material pile in the two side-surfaces according to a rotation direction of the excavator body, and performing a turning on operation on a binocular camera provided on the determined side-surface of the excavator while keeping a binocular camera provided on the other side-surface of the excavator in a turned off sate;
acquiring a sequence of depth images of the material pile collected by the turned on binocular camera provided on the determined side-surface of the excavator; and
performing three-dimensional reconstruction based on the sequence of depth images of the material pile, to generate a three-dimensional model of the material pile.

2. The method according to claim 1 , wherein acquiring a sequence of depth images of the material pile collected by the turned on binocular camera provided on the determined side-surface of the excavator comprises:
acquiring the sequence of depth images of the material pile, the sequence of depth images of the material pile being collected by the binocular camera during the rotation of the excavator body to transport materials shoveled from the material pile, wherein the rotation of the excavator is with a vertical line perpendicular to the ground as an axis and the front-side of the excavator body turns away from the material pile during rotation of the excavator body.

3. The method according to claim 2 , wherein the method further comprises:
in response to detecting an instruction for controlling a bucket of the excavator to unload materials, turning off the binocular camera provided on the determined side-surface of the excavator.

4. The method according to claim 1 , wherein the method further comprises:
excavating, by the excavator, the material pile based on the generated three-dimensional model of the material pile.

5. The method according to claim 4 , wherein the excavating, by the excavator, the material pile based on the generated three-dimensional model of the material pile comprises:
planning, according to the three-dimensional model of the material pile, an excavation operation for excavating the material pile; and
excavating, by the excavator, the material pile according to the generated excavation operation.

6. The method according to claim 5 , wherein excavation operation for excavating the material pile includes at least one of:
an excavating point, an excavating angle and an excavating trajectory.

7. The method according to claim 1 , wherein the performing three-dimensional reconstruction based on the sequence of depth images of the material pile, to generate a three-dimensional model of the material pile comprises:
converting the sequence of depth images of the material pile into a sequence of three-dimensional point clouds;
fusing the sequence of three-dimensional point clouds based on overlaps of adjacent images in the sequence of depth images, to obtain a fused three-dimensional point cloud of the material pile; and
generating the three-dimensional model of the material pile based on the three-dimensional point cloud of the material pile.

8. A three-dimensional reconstruction apparatus for a material pile, applied to an unmanned autonomous excavator, where a binocular camera is provided on each of two side-surfaces of the excavator, and the two side-surfaces of the excavator are parallel to an advancing direction of the excavator, the apparatus comprising:
at least one processor; and
a memory storing instructions, the instructions when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising:
in response to detecting an instruction for controlling an excavator body of the excavator to rotate to transport materials shoveled from the material pile, determining a side-surface turned to the material pile in the two side-surfaces according to a rotation direction of the excavator body, and performing a turning on operation on a binocular camera provided on the determined side-surface of the excavator while keeping a binocular camera provided on the other side-surface of the excavator in a turned off sate;
acquiring a sequence of depth images of the material pile collected by the turned on binocular camera provided on the determined side-surface of the excavator; and
performing three-dimensional reconstruction based on the sequence of depth images of the material pile to, generate a three-dimensional model of the material pile.

9. The apparatus according to claim 8 , wherein the acquiring a sequence of depth images of the material pile collected by the turned on binocular camera provided on the determined side-surface of the excavator comprises:
acquiring the sequence of depth images of the material pile, the sequence of depth images of the material pile being collected by the binocular camera during the rotation of the excavator body to transport materials shoveled from the material pile, wherein the rotation of the excavator is with a vertical line perpendicular to the ground as an axis and the front-side of the excavator body turns away from the material pile during rotation of the excavator body.

10. The apparatus according to claim 9 , wherein the operations further comprise:
in response to detecting an instruction for controlling a bucket of the excavator to unload materials, turning off the binocular camera provided on the determined side-surface of the excavator.

11. The apparatus according to claim 8 , wherein the operations further comprise:
excavating, by the excavator, the material pile based on the generated three-dimensional model of the material pile.

12. The apparatus according to claim 8 , wherein the performing three-dimensional reconstruction based on the sequence of depth images of the material pile, to generate a three-dimensional model of the material pile comprises:
converting the sequence of depth images of the material pile into a sequence of three-dimensional point clouds;
fusing the sequence of three-dimensional point clouds based on overlaps of adjacent images in the sequence of depth images, to obtain a fused three-dimensional point cloud of the material pile; and
generating the three-dimensional model of the material pile based on the three-dimensional point cloud of the material pile.

13. The apparatus according to claim 8 , wherein the excavating, by the excavator, the material pile based on the generated three-dimensional model of the material pile comprises:
planning, according to the three-dimensional model of the material pile, an excavation operation for excavating the material pile; and
excavating, by the excavator, the material pile according to the generated excavation operation.

14. A computer-readable medium, storing a computer program thereon, wherein the program, when executed by a processor, causes the processor to perform operations applied to an unmanned autonomous excavator, where a binocular camera is provided on each of two side-surfaces of the excavator, and the two side-surfaces of the excavator are parallel to an advancing direction of the excavator, the operations comprising:
in response to detecting an instructio

Assignees

• Beijing Baidu Netcom Sci & Tech Co Ltd

Inventors

• Cheng Xinjing
• Yang Ruigang
• Lu Feixiang
• Yang Yajue
• Xu Hao

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• E02F3/439

  Automatic repositioning of the implement, e.g. automatic dumping, auto-return (E02F3/438 takes precedence) · CPC title

• E02F9/262

  with follow-up actions to control the work tool, e.g. controller · CPC title

• E02F3/32

  working downwardly and towards the machine, e.g. with backhoes · CPC title

• G06T15/08 Primary

  Volume rendering · CPC title