Three-dimensional scene constructing method, apparatus and system, and storage medium

Patent metadata

Field	Value
Publication number	US-11954813-B2
Application number	US-202117789217-A
Country	US
Kind code	B2
Filing date	Sep 2, 2021
Priority date	Oct 30, 2020
Publication date	Apr 9, 2024
Grant date	Apr 9, 2024

Abstract

A three-dimensional scene constructing method, apparatus and system, and a storage medium. The three-dimensional scene constructing method includes: acquiring point cloud data of a key object and a background object in a target scene, wherein the point cloud data of the key object comprises three-dimensional information and corresponding feature information, and the point cloud data of the background object at least comprises three-dimensional information; establishing a feature database of the target scene, wherein the feature database at least comprises a key object feature library for recording three-dimensional information and feature information of the key object; performing registration and fusion on the point cloud data of the key object and the point cloud data of the background object, so as to obtain a three-dimensional model of the target scene; and when updating the three-dimensional model, reconstructing the three-dimensional model in a regional manner according to the feature database.

First claim

The invention claimed is:

1. A method for constructing a three-dimensional scene, comprising:
acquiring point cloud data of a key object and a background object in a target scene, wherein the point cloud data of the key object comprises three-dimensional information and corresponding feature information, and the point cloud data of the background object at least comprises three-dimensional information; establishing a feature database of the target scene, wherein the feature database at least comprises a key object feature library for recording three-dimensional information and feature information of the key object;
performing registration and fusion on the point cloud data of the key object and the point cloud data of the background object, so as to obtain a three-dimensional model of the target scene; and
when updating the three-dimensional model, reconstructing the three-dimensional model according to the feature database,
wherein the feature database further comprises an edge region feature library and a mask region feature library;
after performing registration and fusion on the point cloud data of the key object and the point cloud data of the background object, the method further comprises:
determining a peripheral edge region of the key object, acquiring three-dimensional information and feature information of the peripheral edge region, and storing the three-dimensional information of the peripheral edge region, corresponding feature information of the peripheral edge region, and the key object corresponding to the peripheral edge region in the edge region feature library;
estimating three-dimensional information and feature information of a mask region shielded by the key object based on the edge region feature library, storing three-dimensional information of a mask region, corresponding feature information of the mask region, and the key object corresponding to the mask region in the mask region feature library; and
updating the three-dimensional model of the target scene based on the mask region feature library.

2. The three-dimensional scene construction method according to claim 1 , wherein the estimating the three-dimensional information and feature information of the mask region shielded by the key object based on the edge region feature library comprises:
for any mask region, determining the key object corresponding to the mask region;
determining the three-dimensional information and the feature information of the peripheral edge region corresponding to the key object according to the edge region feature library;
determining change rules of the three-dimensional information and the feature information according to the three-dimensional information and the feature information of the peripheral edge region; and
estimating the three-dimensional information and the feature information of the mask region according to the change rules of the three-dimensional information and the feature information.

3. The three-dimensional scene construction method according to claim 2 , wherein determining the change rules of the three-dimensional information and the feature information according to the three-dimensional information and the feature information of the peripheral edge region comprises:
determining a plurality of sampling points on an outer boundary of the mask region, wherein the outer boundary of the mask region is overlapped with an inner boundary of the peripheral edge region;
determining three-dimensional information and feature information of each sampling point according to the three-dimensional information and the feature information of the peripheral edge region;
for each sampling point, searching an image region with similar feature information as the sampling point in a direction away from the mask region; and
acquiring a three-dimensional information change gradient and a feature information change gradient in the image region.

4. The three-dimensional scene construction method according to claim 3 , further comprising:
when a three-dimensional position and/or the feature information of the key object in the target scene image is changed, acquiring the point cloud data of the key object at a changed place and updating the three-dimensional model.

5. The three-dimensional scene construction method according to claim 2 , further comprising:
when a three-dimensional position and/or the feature information of the key object in the target scene image is changed, acquiring the point cloud data of the key object at a changed place and updating the three-dimensional model.

6. The three-dimensional scene construction method according to claim 2 , wherein reconstructing the three-dimensional model according to the feature database when updating the three-dimensional model comprises:
based on the feature database, acquiring part or all dense point cloud data of one or a plurality of regions every other frame or a plurality of frames of target scene images, and performing dense reconstruction on the three-dimensional model with the dense point cloud data.

7. The three-dimensional scene construction method according to claim 1 , further comprising:
when a three-dimensional position and/or the feature information of the key object in the target scene image is changed, acquiring the point cloud data of the key object at a changed place and updating the three-dimensional model.

8. The three-dimensional scene construction method according to claim 7 , wherein acquiring the point cloud data of the key object at the changed place comprises:
acquiring the point cloud data at the change place of the key object through an image acquisition apparatus; or,
using the three-dimensional information of the key object at the change place to update the point cloud data of the key object; or
predicting the change place of the key object by using motion estimation or motion compensation, calculating the point cloud data of the change place in advance, and updating the three-dimensional model by using the point cloud data of the change place.

9. The three-dimensional scene construction method according to claim 8 , wherein after predicting the change place of the key object by using motion estimation or motion compensation and calculating the point cloud data of the change place in advance, the method further comprises:
acquiring a current actual point cloud data of the key object in the target scene, comparing the current actual point cloud data with the pre-calculated point cloud data, determining point cloud data of the difference part, and re-updating the three-dimensional model with the point cloud data of the difference part.

10. The three-dimensional scene construction method according to claim 1 , wherein reconstructing the three-dimensional model according to the feature database when updating the three-dimensional model comprises:
based on the feature database, acquiring part or all dense point cloud data of one or a plurality of regions every other frame or a plurality of frames of target scene images, and performing dense reconstruction on the three-dimensional model with the dense point cloud data.

11. The three-dimensional scene construction method according to claim 10 , further comprising:
estimating point cloud data of a mask region using the point cloud data of a dense reconstructed edge region, and updating the three-dimensional model.

12. The three-dimensional scene construction method according to claim 10 , further comprising:
estimating point cloud data of adjacent positions by using acquired dense point cloud data.

13. The three-dimensional scene construction method according to claim 1 , wherein the feature

Assignees

• Boe Technology Group Co Ltd

Inventors

• Wang Youxue
• Ma Xiaohui
• Geng Kai
• Hou Mengjun
• Ha Qian

Classifications

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T7/12

  Edge-based segmentation · CPC title

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T2219/2004

  Aligning objects, relative positioning of parts · CPC title

• H04N13/275 Primary

  from three-dimensional [3D] object models, e.g. computer-generated stereoscopic image signals · CPC title