Rendering augmented reality image including virtual object with surface showing reflection of environment

Patent metadata

Field	Value
Publication number	US-11308701-B2
Application number	US-202016862990-A
Country	US
Kind code	B2
Filing date	Apr 30, 2020
Priority date	Oct 31, 2017
Publication date	Apr 19, 2022
Grant date	Apr 19, 2022

Abstract

In accordance with an aspect of the present disclosure, there is provided a method for processing image performed by image processing device for rendering an augmented reality image. The method comprises, acquiring a first real image corresponding to a first direction from the image processing device, acquiring a second real image corresponding to a second direction, different from the first direction, from the image processing device, determining reflectivity of a surface of a virtual object in the augmented reality image; and rendering, according to the reflectivity, the surface of the virtual object by using an environment texture, the environment texture generated based on the first real image and the second real image.

First claim

What is claimed is:

1. A method for processing an image performed by an image processing device for rendering an augmented reality image, the method comprising:
acquiring a first real image corresponding to a first direction from the image processing device;
acquiring a second real image corresponding to a second direction, different from the first direction, from the image processing device;
determining reflectivity of a surface of a virtual object in the augmented reality image; and
rendering, according to the reflectivity, the surface of the virtual object by using an environment texture, the environment texture generated by (i) projecting the first real image onto a center area of the virtual object, and (ii) projecting the second real image onto an edge area of the virtual object.

2. The method of claim 1 , wherein the second direction is a direction opposite to the first direction.

3. The method of claim 1 , wherein the rendering the surface of the virtual object includes,
setting an image connection area including sides where at least one side of the first real image and at least one side of the second real image abut; and
diffusing color of the image connection area to the first real image and the second real image.

4. The method of claim 3 , wherein the diffusing the color of the image connection area comprises determining color to be diffused based on a Poisson Equation where the color of the image connection area is provided as an input to the Poisson Equation, or based on a Blur operation applied to the color of the image connection area.

5. The method of claim 3 , wherein the setting the image connection area includes,
generating a first mirror image by mirroring the first real image along a left side or a right side of the first real image as a vertical axis;
generating a second mirror image by mirroring the first real image along a top side or a bottom side of the first real image as a horizontal axis; and
setting a plurality of image connection areas including each side of the second real image, and
wherein each side of the second real image abuts a side of one of (i) the first real image, (ii) the first mirror image and (iii) the second mirror image.

6. The method of claim 3 , wherein the setting the image connection area includes,
generating a third mirror image by mirroring the second real image along a left side or a right side of the second real image as a vertical axis;
generating a fourth mirror image by mirroring the second real image along a top side or a bottom side of the second real image as a horizontal axis; and
setting a plurality of image connection areas including each side of the first real image, and
wherein each side of the first real image abuts a side of one of (i) the second real image, (ii) the third mirror image and (iii) the fourth mirror image.

7. The method of claim 1 , wherein the rendering the surface of the virtual object includes,
generating a reality modified image rendered by overlaying an avatar of a user on the second real image when an image of the user is included among the real objects in the second real image; and
determining the first real image and the reality modified image as the environment texture.

8. A non-transitory computer-readable storage medium including computer executable instructions, wherein the instructions, when executed by a processor, cause the processor to perform a method for processing an image for rendering an augmented reality image, the method comprising:
acquiring a first real image corresponding to a first direction;
acquiring a second real image corresponding to a second direction, different from the first direction;
determining reflectivity of a surface of a virtual object in the augmented reality image; and
rendering, according to the reflectivity, the surface of the virtual object by using an environment texture, the environment texture generated by (i) projecting the first real image onto a center area of the virtual object, and (ii) projecting the second real image onto an edge area of the virtual object.

9. An image processing device, comprising:
a processor; and
memory storing instructions thereon, the instructions when executed by the processor cause the processor to:
acquire a first real image corresponding to a first direction from the image processing device and a second real image corresponding to a second direction, different from the first direction, from the image processing device;
astore three-dimensional model data corresponding to a virtual object in an augmented reality image;
determine, for the virtual object in the augmented reality image, reflectivity of a surface of the virtual object based on the three-dimensional model data; and
render, according to the reflectivity, the surface of the virtual object by using an environment texture, the environment texture generated by (i) projecting the first real image onto a center area of the virtual object, and (ii) projecting the second real image onto an edge area of the virtual object.

Assignees

• Sk Telecom Co Ltd

Inventors

• Shin Seungho
• Jeon Jinsoo
• Lim Gukchan
• Cho Ikhwan

Classifications

• G06T7/90

  Determination of colour characteristics · CPC title

• G06T15/04 Primary

  Texture mapping · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T7/529

  from texture · CPC title

• G06T15/50

  Lighting effects · CPC title