Augmented reality scene image processing method and apparatus, electronic device and storage medium

Patent metadata

Field	Value
Publication number	US-11423625-B2
Application number	US-202017134811-A
Country	US
Kind code	B2
Filing date	Dec 28, 2020
Priority date	Oct 15, 2019
Publication date	Aug 23, 2022
Grant date	Aug 23, 2022

Abstract

An Augmented Reality (AR) scene image processing method, an electronic device and a storage medium are provided. The method includes that: shooting pose data of an AR device is acquired; presentation special effect data of a virtual object corresponding to the shooting pose data in a reality scene is acquired based on the shooting pose data and position pose data of the virtual object in a three-dimensional scene model representing the reality scene; and an AR scene image is displayed through the AR device based on the presentation special effect information.

First claim

The invention claimed is:

1. An Augmented Reality (AR) scene image processing method, comprising:
acquiring shooting pose data of an AR device;
acquiring presentation special effect information of a virtual object corresponding to the shooting pose data in a reality scene based on the shooting pose data and pose data of the virtual object in a three-dimensional scene model configured to represent the reality scene; and
displaying an AR scene image through the AR device based on the presentation special effect information,
wherein the three-dimensional scene model is generated in the following manner:
acquiring multiple reality scene images corresponding to the reality scene; and
generating the three-dimensional scene model based on the multiple reality scene images;

wherein generating the three-dimensional scene model based on the multiple reality scene images comprises:
extracting multiple feature points from each reality scene image of the multiple reality scene images; and
generating the three-dimensional scene model based on the multiple feature points and a pre-stored three-dimensional sample image matched with the reality scene, wherein the pre-stored three-dimensional sample image comprises a pre-stored three-dimensional image representing a morphology feature of the reality scene.

2. The method of claim 1 , wherein acquiring the presentation special effect information of the virtual object corresponding to the shooting pose data in the reality scene based on the shooting pose data and the pose data of the virtual object in the three-dimensional scene model configured to represent the reality scene comprises:
acquiring the presentation special effect information of the virtual object corresponding to the shooting pose data based on the shooting pose data, the pose data of the virtual object in the three-dimensional scene model, and the three-dimensional scene model.

3. The method of claim 1 , wherein acquiring the shooting pose data of the AR device comprises:
acquiring a reality scene image shot by the AR device; and
determining shooting pose data corresponding to the reality scene image based on the reality scene image and a pre-stored first neural network model for positioning, wherein the shooting pose data corresponding to the reality scene image comprises at least one of shooting position information or shooting orientation information.

4. The method of claim 3 , wherein the pre-stored first neural network model is trained according to the following step:
training the pre-stored first neural network model based on multiple sample images obtained by shooting of the reality scene in advance and shooting pose data corresponding to each of the multiple sample images.

5. The method of claim 1 , wherein acquiring the shooting pose data of the AR device comprises:
acquiring a reality scene image shot by the AR device; and
determining shooting pose data corresponding to the reality scene image based on the reality scene image and an aligned three-dimensional sample image, wherein the shooting pose data corresponding to the reality scene image comprises at least one of shooting position information or shooting orientation information, and the aligned three-dimensional sample image is a three-dimensional sample image obtained after feature point alignment of a sample image library obtained by shooting of the reality scene in advance and the pre-stored three-dimensional sample image.

6. The method of claim 5 , wherein determining the shooting pose data corresponding to the reality scene image based on the reality scene image and the aligned three-dimensional sample image comprises:
determining a feature point, matched with a feature point in the reality scene image, in the three-dimensional sample image based on the aligned three-dimensional sample image;
determining a target sample image matched with the reality scene image in the sample image library based on coordinate information of the feature point in the three-dimensional sample image in the aligned three-dimensional sample image, wherein the sample image library comprises multiple sample images obtained by shooting of the reality scene in advance and shooting pose data corresponding to each of the multiple sample images; and
determining the shooting pose data corresponding to the target sample image as the shooting pose data corresponding to the reality scene image.

7. The method of claim 1 , wherein after acquiring the shooting pose data of the AR device, the method further comprises:
acquiring a reality scene image shot by the AR device; and
determining attribute information corresponding to the reality scene image based on the reality scene image and a pre-stored second neural network model that is configured to determine the attribute information corresponding to the reality scene image, wherein
acquiring the presentation special effect information of the virtual object corresponding to the shooting pose data in the reality scene based on the shooting pose data and the pose data of the virtual object in the three-dimensional scene model configured to represent the reality scene comprises:
acquiring the presentation special effect information of the virtual object corresponding to the shooting pose data in the reality scene based on the shooting pose data, the attribute information, and the pose data of the virtual object in the three-dimensional scene model configured to representing the reality scene.

8. The method of claim 7 , wherein the pre-stored second neural network model is trained according to the following step:
training the pre-stored second neural network model based on multiple sample images obtained by shooting of the reality scene in advance and attribute information corresponding to each of the multiple sample images.

9. The method of claim 1 , wherein after acquiring the shooting pose data of the AR device, the method further comprises:
acquiring a preset identifier of a reality scene shot by the AR device; and
determining additional virtual object information corresponding to the reality scene shot by the AR device based on the preset identifier and a pre-stored mapping relationship between preset identifiers and the additional virtual object information,
wherein acquiring the presentation special effect information of the virtual object corresponding to the shooting pose data in the reality scene based on the shooting pose data and the pose data of the virtual object in the three-dimensional scene model configured to represent the reality scene comprises:
acquiring the presentation special effect information of the virtual object corresponding to the shooting pose data in the reality scene based on the shooting pose data, the additional virtual object information, and the pose data of the virtual object in the three-dimensional scene model configured to represent the reality scene.

10. The method of claim 1 , wherein after displaying the AR scene image through the AR device based on the presentation special effect information, the method further comprises:
acquiring a triggering operation for the virtual object displayed in the AR device, and updating the presentation special effect information presented in the AR scene image.

11. The method of claim 10 , wherein the virtual object comprises a target musical instrument; and acquiring the triggering operation for the virtual object displayed in the AR device and updating the presentation special effect information presented in the AR scene image comprises:
acquiring the triggering operation for the virtual object displayed in the AR device, and controlling the AR device to update a sound playing effect of the virtual object to

Assignees

• Beijing Sensetime Tech Development Co Ltd

Inventors

• Hou Xinru
• Luan Qing
• Sheng chongshan
• Jiao Fei
• Ou Huafu
• Shi Shengchuan
• Wang Nan
• Jiang Hanqing

Classifications

• G06V20/35

  Categorising the entire scene, e.g. birthday party or wedding scene · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title