Data processing method and device used in virtual scenario

Patent metadata

Field	Value
Publication number	US-12002239-B2
Application number	US-202117230102-A
Country	US
Kind code	B2
Filing date	Apr 14, 2021
Priority date	Oct 15, 2018
Publication date	Jun 4, 2024
Grant date	Jun 4, 2024

Abstract

This application provides a data processing method and device. The method includes: collecting a first image by using a camera; obtaining first point cloud data of the object based on the first image, where the first point cloud data indicates three-dimensional coordinate values of a surface point of the object in a first camera coordinate system of the first terminal; obtaining pose data of the first terminal in the first camera coordinate system based on the first image; obtaining a two-dimensional feature image based on the pose data, the first point cloud data, and the first image; generating map data based on the first point cloud data, the pose data, and the two-dimensional feature image; and sending the map data to a second terminal or a server.

First claim

What is claimed is:

1. A data processing method used in a virtual scenario, comprising:
collecting, by a first terminal, a first image by using a camera, wherein the first image comprises an image of an object;
obtaining, by the first terminal, first point cloud data of the object based on the first image, wherein the first point cloud data indicates three-dimensional coordinate values of a surface point of the object in a first camera coordinate system of the first terminal;
obtaining, by the first terminal, pose data of the first terminal in the first camera coordinate system based on the first image;
obtaining, by the first terminal, a two-dimensional feature image based on the pose data, the first point cloud data, and the first image;
generating, by the first terminal, map data based on the first point cloud data, the pose data, and the two-dimensional feature image; and
sending, by the first terminal, the map data to a second terminal or a server;
wherein the obtaining the two-dimensional feature image based on the pose data, the first point cloud data, and the first image comprises:
projecting, by the first terminal, the first point cloud data onto the first image based on the pose data, to obtain two-dimensional image coordinates that are corresponding to the first point cloud data and that are in an image coordinate system of the first image; and
obtaining, by the first terminal, the two-dimensional feature image based on an image block or an image feature point descriptor corresponding to the two-dimensional image coordinates and in the first image.

2. The method of claim 1 , wherein the obtaining pose data of the first terminal in the first camera coordinate system based on the first image comprises:
obtaining, by the first terminal, the pose data of the first terminal in the first camera coordinate system based on the first image and motion data that is collected by an inertial measurement unit IMU.

3. The method of claim 1 , wherein before the generating map data based on the first point cloud data, the pose data, and the two-dimensional feature image, the method further comprises:
obtaining, by the first terminal, positioning data comprising at least one of GPS positioning data of the first terminal, BeiDou positioning data of the first terminal, Wi-Fi positioning data of the first terminal, Bluetooth positioning data of the first terminal, or base station positioning data of the first terminal, wherein
correspondingly, the map data further carries the positioning data.

4. The method of claim 1 , wherein the virtual scenario is one of a virtual reality (VR) scenario, an augmented reality (AR) scenario, and a mixed reality (MR) scenario.

5. A data processing method used in a virtual scenario, comprising:
receiving, by a second terminal, map data from a first terminal, wherein the map data is generated based on first point cloud data of an object, pose data of the first terminal in a first camera coordinate system of the first terminal, and a two-dimensional feature image of the first terminal, wherein the first point cloud data indicates three-dimensional coordinate values of a surface point of the object in the first camera coordinate system of the first terminal;
obtaining, by the second terminal, pose data of the second terminal in a second camera coordinate system of the second terminal;
obtaining, by the second terminal, a two-dimensional feature image of the second terminal;
obtaining, by the second terminal, coordinate system transformation data between the first camera coordinate system and the second camera coordinate system based on the map data, the pose data of the second terminal in the second camera coordinate system, and the two-dimensional feature image of the second terminal; and
synchronizing, by the second terminal, content in the virtual scenario based on the coordinate system transformation data.

6. The method of claim 5 , wherein the obtaining coordinate system transformation data comprises:
performing, by the second terminal, a feature matching operation in the map data using the two-dimensional feature image of the second terminal, to obtain pose data of the second terminal in the first camera coordinate system; and
obtaining, by the second terminal, the coordinate system transformation data between the first camera coordinate system and the second camera coordinate system based on the pose data of the second terminal in the first camera coordinate system and the pose data of the second terminal in the second camera coordinate system.

7. The method of claim 5 , wherein before the obtaining pose data of the second terminal in the second camera coordinate system of the second terminal, the method further comprises:
collecting, by the second terminal, a second image comprising an image of the object; and
obtaining, by the second terminal, second point cloud data of the object based on the second image, wherein the second point cloud data indicates three-dimensional coordinate values of a surface point of the object in the second camera coordinate system;
wherein the obtaining pose data of the second terminal in the second camera coordinate system of the second terminal comprises: obtaining the pose data of the second terminal in the second camera coordinate system based on the second image; and
wherein the obtaining the two-dimensional feature image of the second terminal comprises:
projecting the second point cloud data onto the second image based on the pose data of the second terminal in the second camera coordinate system, to obtain two-dimensional image coordinates corresponding to the second point cloud data and in an image coordinate system of the second image, and
obtaining the two-dimensional feature image of the second terminal based on an image block or an image feature point descriptor corresponding to the two-dimensional image coordinates and in the second image.

8. The method of claim 5 , wherein the synchronizing content in the virtual scenario based on the coordinate system transformation data comprises:
obtaining, by the second terminal, pose transformation data of an anchor in the virtual scenario based on the coordinate system transformation data; and
synchronizing, by the second terminal, the content in the virtual scenario based on the pose transformation data of the anchor.

9. The method of claim 5 , wherein the map data further carries positioning data comprising at least one of GPS positioning data of the first terminal, BeiDou positioning data of the first terminal, Wi-Fi positioning data of the first terminal, Bluetooth positioning data of the first terminal, or base station positioning data of the first terminal; and
wherein the obtaining the two-dimensional feature image of the second terminal comprises:
determining a search scope of the two-dimensional feature image based on the positioning data, and
determining the two-dimensional feature image of the second terminal from a plurality of historical two-dimensional feature images based on the search scope.

10. A data processing terminal used in a virtual scenario, comprising:
a camera, configured to collect a first image comprising an image of an object;
a spatial positioning engine, configured to obtain first point cloud data of the object based on the first image, wherein the first point cloud data indicates three-dimensional coordinate values of a surface point of the object in a first camera coordinate system of the terminal; and obtain pose data of the terminal in the first camera coordinate system based on the first image;
a feature extractor, configured to obtain a two-dimensional feature image based on the pose data, the first point cloud data, and the first

Assignees

• Huawei Tech Co Ltd

Inventors

• Jiang Jun
• Liu Xiaowei
• Feng Weiteng
• Mo Yunneng
• Wu Zongwu

Classifications

• G06T7/74 Primary

  involving reference images or patches · CPC title

• G09G5/12

  Synchronisation between the display unit and other units, e.g. other display units, video-disc players · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T19/00 Primary

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title