Augmented reality scenario generation method, apparatus, system, and device

Patent metadata

Field	Value
Publication number	US-11270419-B2
Application number	US-201916294890-A
Country	US
Kind code	B2
Filing date	Mar 6, 2019
Priority date	Oct 26, 2016
Publication date	Mar 8, 2022
Grant date	Mar 8, 2022

Abstract

Aspects of the disclosure provide a method, an apparatus and a system for augmented reality. In an example, an apparatus includes an imaging device, processing circuitry and a display device. The imaging device generates a first image of a surrounding environment of the apparatus. The processing circuitry processes first image data of the first image and second image data of a second image to fuse the first image and the second image into a fusion image. The second image includes an object that is not in the surrounding environment. The display device then displays the fusion image.

First claim

What is claimed is:

1. A method for augmented reality in an augmented reality game, comprising:
obtaining, by processing circuitry of an apparatus and via an imaging device, first image data of a first image that is taken of a surrounding environment of the apparatus;
obtaining an environment parameter that is indicative of a rendering mode, including an infrared mode, a night vision mode, a cartoon mode, or a sketch mode;
performing, by the processing circuitry, a set of image processing operations associated with the rendering mode on the first image data of the first image to obtain a rendered background image that has a visual effect indicated by the rendering mode;
analyzing the first image to identify a virtual object in the augmented reality game according to at least one element in the first image;
generating, by the processing circuitry executing a game engine of the augmented reality game, a second image that includes a virtual role image of a virtual role and a virtual object image of the virtual object in the augmented reality game, the virtual role representing a user and controllable by the user to interact with the virtual object in the augmented reality game;
combining the rendered background image and the second image to obtain a fusion image;
controlling, by the processing circuitry, a display screen to display the fusion image;
receiving a user operation conesponding to an interaction between the virtual role and the virtual object in the augmented reality game;
updating, by the processing circuitry executing the game engine of the augmented reality game, the second image to obtain an updated second image showing the interaction by at least adjusting the virtual role image according to the user operation; and
updating the fusion image for display according to the updated second image.

2. The method according to claim 1 , further comprising at least one of:
controlling a camera to take the first image of the surrounding environment and provide the first image data of the first image to the processing circuitry; and
controlling a holographic scanner to scan the surrounding enviroment to generate the first image data.

3. The method according to claim 1 , wherein, when the environment parameter is indicative of the infrared mode, the performing the set of image processing operations associated with the rendering mode on the first image data of the first image comprises:
converting the first image into a grayscale image;
determining respective first heat values of first pixels in the grayscale image based on first grayscale values of the first pixels in the grayscale image; and
performing a first color adjustment on the grayscale image based on the first heat values of the first pixels, to obtain the rendered background image, the first color adjustment converting a relatively higher heat value to a color of a relatively higher color temperature.

4. The method according to claim 1 , when the environment parameter is indicative of the infrared mode, the performing the set of image processing operations associated with the rendering mode on the first image data of the first image comprising:
converting the first image into a grayscale image;
scaling down the grayscale image based on a scale to obtain a temporary image;
determining respective second heat values of second pixels in the temporary image based on second grayscale values of the second pixels in the temporary image;
performing a second color adjustment on the temporary image based on the second heat values of the second pixels to obtain a color-adjusted temporary picture; and
scaling up the color-adjusted temporary picture based on the scale, to obtain the rendered background image.

5. The method according to claim 1 , wherein, when the environment parameter is indicative of the night vision mode, the performing the set of image processing operations associated with the rendering mode on the first image data of the first image comprises:
obtaining a central pixel of the first image;
calculating distances between pixels in the first image and the central pixel;
setting respective distortion degrees of the pixels based on the distances to obtain a distorted image, the distortion degrees being positively correlated to the distances;
green-lens processing the distorted image to obtain a greened image; and
adding a random noise to the greened image to generate the rendered background image.

6. The method according to claim 1 , further comprising:
receiving rotation data that is detected by a gyroscope; and
updating the second image by adjusting the virtual role image to reflect a rotation of the virtual role based on the rotation data.

7. The method according to claim 1 , wherein the obtaining the environment parameter comprises:
determining the environment parameter according to an analysis of the first image.

8. An apparatus, comprising:
an imaging device configured to generate first image data of a first image that is taken of a surrounding environment of the apparatus; and
processing circuitry configured to
obtain an environment parameter that is indicative of a rendering mode, including an infrared mode, a night vision mode, a cartoon mode, or a sketch mode,
perform a set of image processing operations associated with the rendering mode on the first image data of the first image to obtain a rendered background image that has a visual effect indicated by the rendering mode,
analyze the first image to identify a virtual object in an augmented reality game according to at least one element in the first image;
generate, by a game engine of an augmented reality game, a second image that includes a virtual role image of a virtual role and a virtual object image of the virtual object in the augmented reality game, the virtual role representing a user and controllable by the riser to interact with the virtual object in the augmented reality game,
combine the rendered backgroumd image and the second image to obtain a fusion image,
receive a user operation corresponding to an interaction between the virtual role and the virtual object in the augmented reality game,
update, by the game engine of the augmented reality game, the second image to obtain an updated second image showing the interaction by at least adjusting the virtual role image according to the user operation, and
update the fusion image for display according to the updated second image; and

a display device configured to display the fusion image.

9. The apparatus according to claim 8 , wherein the imaging device comprises at lust one of:
a camera configured to take the first image of the surrounding environment; and
a holographic scanner configured to scan the surrounding environment to generate the first image data of the first image.

10. The apparatus according to claim 8 , wherein the processing circuitry is configured to:
when the environment parameter is indicative of the infrared mode,
convert the first image into a grayscale image,
determine respective first heat values of first pixels in the grayscale image based on first grayscale values of the first pixels in the grayscale image, and
perform a first color adjustment on the grayscale image based on the first heat values of the first pixels to obtain the rendered back round image, the first color adjustment converting a relatively higher heat value to a color of a relatively higher color temperature.

11. The apparatus according to claim 8 , wherein the processing circuitry is configured to:
when the environment parameter is indicative of the infrared mode,
convert the first image into a grayscale image,
scale down the gayscale image based on a

Assignees

• Tencent Tech Shenzhen Co Ltd

Inventors

• Wu Yanlin
• Li Deyuan
• Wang Peng

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• A63F13/537

  using indicators, e.g. showing the condition of a game character on screen · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• A63G31/00

  Amusement arrangements · CPC title