Terrain modeling method that fuses geometric characteristics and mechanical characteristics and computer readable storage medium thereof

Patent metadata

Field	Value
Publication number	US-11195329-B2
Application number	US-202016826326-A
Country	US
Kind code	B2
Filing date	Mar 23, 2020
Priority date	Jun 21, 2019
Publication date	Dec 7, 2021
Grant date	Dec 7, 2021

Abstract

A terrain modeling method that fuses geometric characteristics and mechanical characteristics, and a terrain modeling system are provided in the present invention. The method includes steps of: obtaining color images and depth images of the detected region, performing a terrain semantic segmentation on the color images, and fusing ground semantic information obtained by the semantic segmentation and depth information contained in the depth images at the same time to generate the point clouds; mapping the point clouds to a raster map in a map coordinate system to generate corresponding rasters, and updating the elevation values to the corresponding rasters; and calculating the input images in terms of terra-mechanical characteristics, and updating calculation results to the corresponding rasters to generate a terrain model. A mechanical characteristic parameter is added to a terrain factor, and terrain characteristics are innovatively performed from two dimensions of geometric characteristic and mechanical characteristics.

First claim

What is claimed is:

1. A terrain modeling method that fuses geometric characteristics and mechanical characteristics, comprising steps of:
obtaining input images of a detected region, and processing the input images to generate point clouds comprising elevation values of the input images;
wherein the step of obtaining the input images of the detected region and processing the input images to generate point clouds comprises:
obtaining color images and depth images of the detected region, performing a terrain semantic segmentation on the color images, and fusing ground semantic information obtained by the semantic segmentation and depth information contained in the depth images at the same time to generate the point clouds;

mapping the point clouds to a raster map in a map coordinate system to generate corresponding rasters, and updating the elevation values to the corresponding rasters; and
calculating the input images in terms of terra-mechanical characteristics, and updating calculation results to the corresponding rasters to generate a terrain model.

2. The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 1 , wherein the step of performing a terrain semantic segmentation on the color images comprises:
collecting terrain images as a data source, annotating a target terrain in the collected terrain images and generating corresponding mask labels, and randomly dividing the annotated terrain images into training sets and validation sets in proportion;
calculating a mean and a variance of three RGB channels of the training sets, respectively, performing a data normalization on each of the input images, performing a data enhancement on original images in the training sets, and correcting the corresponding mask labels; and
constructing a convolutional neural network, performing a training on a semantic segmentation model, and selecting a model with the highest accuracy as the semantic segmentation model for performing the terrain semantic segmentation on the color images.

3. The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 1 , wherein the semantic information comprises a plurality of terrain types with a higher confidence obtained after the terrain semantic segmentation and corresponding probabilities.

4. The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 1 , wherein the step of fusing ground semantic information obtained by the semantic segmentation and depth information contained in the depth images at the same time to generate the semantic point clouds comprises:
combining the semantic information of each pixel in the color images with the depth information contained in the depth images at the same time; and
performing a coordinate transformation using pose information of a camera carried by a planet rover to generate the semantic point clouds in the map coordinate system.

5. The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 4 , wherein the step of performing a coordinate transformation using pose information of a camera carried by a planet rover to generate the semantic point clouds in the map coordinate system comprises:
mapping three-dimensional point clouds in a depth camera coordinate system to a color camera coordinate system at the same time to form color three-dimensional point clouds in the color camera coordinate system; and
transforming the color three-dimensional point clouds into the map coordinate system via a planet rover base coordinate system.

6. The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 5 , wherein the color camera coordinate system and the depth camera coordinate system are fixedly connected to a planet rover body, and the planet rover base coordinate system changes with changes in poses of the planet rover and has a projection of a coordinate origin on an XY plane of the map coordinate system coinciding with a origin of the map coordinate system.

7. The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 1 , wherein the step of mapping the point clouds to a raster map in a map coordinate system to generate corresponding rasters, and assigning the elevation values to the corresponding rasters comprises:
updating the elevation values and the semantic information contained in the rasters through a data fusion when the point clouds at different times correspond to the same raster.

8. The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 7 , wherein the elevation values are updated by obtaining an average value of the elevation values contained in the plurality of point clouds.

9. The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 7 , wherein the step of updating the semantic information through a data fusion comprises:
updating the terrain types contained in the plurality of point clouds and the probabilities through Bayesian updating rule.

10. The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 7 , wherein the step of updating the semantic information through a data fusion comprises:
remaining the updated terrain types and a corresponding updated probability being an average of the two probabilities if the corresponding types of the maximum probability of the two semantic point clouds are the same; and
remaining the terrain types with a larger probability as the updated terrain types and a corresponding updated probability being a times the corresponding larger probability if the corresponding types of the maximum probability of the two semantic point clouds are different.

11. The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 1 , wherein the step of estimating the input images in terms of terra-mechanical characteristics, and updating calculation results to the corresponding rasters to generate a terrain model comprises:
fitting a distribution function according to statistical data of the terra-mechanical characteristics; and
performing a numerical estimation of the terra-mechanical characteristics with safety factors in combination with the semantic information, and assigning estimation results to the corresponding rasters to generate the terrain model.

12. The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 11 , wherein the step of performing a numerical calculation of the terra-mechanical characteristics with safety factors in combination with the semantic information comprises:
calculating safety factors S(Ks) of equivalent stiffness modulus Ks and safety factors S(φ) of equivalent friction modulus (φ, respectively:

S
⁡

(

K
s

)

=

∑
i

⁢

c
i

⁢

Assignees

• Harbin Inst Technology

Inventors

• Ding Liang
• Deng Zongquan
• Zhou Ruyi
• Gao Haibo
• Feng Wenhao
• Xu Peng
• Li Nan
• Liu Zhen

Classifications

• G06V20/38

  Outdoor scenes · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T17/05 Primary

  Geographic models · CPC title

• G06N3/08 Primary

  Learning methods · CPC title