Apparatus and method for providing projection mapping-based augmented reality

Patent metadata

Field	Value
Publication number	US-2017200313-A1
Application number	US-201615241543-A
Country	US
Kind code	A1
Filing date	Aug 19, 2016
Priority date	Jan 7, 2016
Publication date	Jul 13, 2017
Grant date	—

Abstract

An apparatus and method for providing projection mapping-based augmented reality (AR). According to an exemplary embodiment, the apparatus includes an input to acquire real space information and user information; and a processor to recognize a real environment by using the acquired real space information and the acquired user information, map the recognized real environment to a virtual environment, generate augmented content that changes corresponding to a change in space or a user's movement, and project and visualize the generated augmented content through a projector.

First claim

What is claimed is:

1 . An apparatus for providing augmented reality (AR), the apparatus comprising:
an input configured to acquire real space information and user information; and a processor configured to recognize a real environment by using the acquired real space information and the acquired user information, map the recognized real environment to a virtual environment, generate augmented content that changes corresponding to a change in space or a user's movement, and project and visualize the generated augmented content through a projector.

2 . The apparatus of claim 1 , wherein the input is configured to acquire in advance the user information comprising a user's skeleton information and body information of each body part; and
the processor is configured to use the user information so that when the augmented content is projected to a user's body, the augmented content matches the user's body.

3 . The apparatus of claim 1 , wherein the input is configured to:
acquire point cloud information of three-dimensional space with regard to real space in which a user and a three-dimensional item model are all removed, match the point cloud information to a three-dimensional background model that is made in advance to be simplified, and register the matched information; and acquire an image and a depth information map regarding each three-dimensional item model used in the augmented content, as well as point cloud information that is made using the image and the depth information map, match the point cloud information to the three-dimensional background model, and register the matched information.

4 . The apparatus of claim 1 , wherein the processor comprises:
an interaction processor configured to recognize an object by using the real space information and the user information, recognize the real environment comprising the user's movement from the recognized object, calculate an interaction between the recognized real environment and the virtual environment, combine the virtual environment with the real environment, and accordingly generate the augmented content, and a projection visualizer configured to project and visualize the augmented content, generated by the interaction processor, through the projector.

5 . The apparatus of claim 4 , wherein the interaction processor is configured to recognize the object by analyzing real space through image processing and machine learning which are performed based on the real space information comprising depth information and point cloud information.

6 . The apparatus of claim 4 , wherein the interaction processor is configured to:
calculate the interaction between real space and virtual space, divide space by using a three-dimensional background model that is made in advance and simplified in order to improve a reaction speed, perform pre-matching for each divided space, and search for an area, which is good enough for the object to be added to, on space where the augmented content is to be represented.

7 . The apparatus of claim 4 , wherein the projection visualizer is configured to acquire mapping parameters between real space and virtual space, and combine the mapping parameters so that the real space and the virtual space are mapped equally.

8 . The apparatus of claim 4 , wherein the projection visualizer is configured to represent the augmented content by training and registering a three-dimensional background model that is made in advance by the input and simplified, searching for an object location on space, where the augmented content is to be represented, by using data acquired by the input, and replacing the searched object location with a virtual object mesh that is made in advance and simplified.

9 . The apparatus of claim 4 , wherein the projection visualizer is configured to in response to the augmented content being projected to a user's body, render a virtual object mesh in three-dimensional space without any change by using user body information acquired in advance by the input, wherein the virtual object mesh is made in advance and simplified.

10 . The apparatus of claim 4 , wherein the projection visualizer is configured to perform edge blending and masking on an image to process an area overlapped by several projectors.

11 . The apparatus of claim 4 , wherein the processor further comprises:
a content sharing processor configured to share and synchronize the augmented content with other users existing in remote areas so that the users experience the augmented content together.

12 . The apparatus of claim 4 , wherein the processor further comprises:
a content logic processor configured to support the augmented content to progress according to a scenario logic, and provide augmented content visualization data to the projection visualizer.

13 . A method of providing AR, the method comprising:
acquiring real space information and user information; recognizing an object by using the acquired real space information and the acquired user information, recognizing a real environment comprising a user's movement from the recognized object, calculating an interaction between the recognized real environment and a virtual environment, combining the virtual environment with the real environment, and accordingly generating augmented content; and projecting and visualizing the generated augmented content through a projector.

14 . The method of claim 13 , wherein the acquiring of the real space information and the user information comprises:
acquiring point cloud information of three-dimensional space with regard to real space in which a user and a three-dimensional item model are all removed, matching the point cloud information to a three-dimensional background model that is made in advance to be simplified, and registering the matched information; and acquiring an image and a depth information map regarding each three-dimensional item model used in the augmented content, as well as point cloud information that is made using the image and the depth information map, matching the point cloud information to the three-dimensional background model, and registering the matched information.

15 . The method of claim 13 , wherein the generating of the augmented content comprises recognizing an object by analyzing real space through image processing and machine learning, which are performed based on the real space information comprising depth information and point cloud information.

16 . The method of claim 13 , wherein the generating of the augmented content comprises:
calculating the interaction between real space and virtual space, dividing space by using a three-dimensional background model that is made in advance and simplified in order to improve a reaction speed, performing pre-matching for each divided space, and searching for an area, which is good enough for the object to be added to, on space where the augmented content is to be represented.

17 . The method of claim 13 , wherein the generating of the augmented content comprises:
acquiring mapping parameters between real space and virtual space, and combining the mapping parameters together so that the real space and the virtual space are mapped equally.

18 . The method of claim 13 , wherein the generating of the augmented content comprises:
representing the augmented content by training and registering a three-dimensional background model that is made in advance and simplified, by searching for an object location on space, where the augmented content is to be represented, using the real space information and

Assignees

• Electronics & Telecommunications Res Inst

Inventors

• Lee Ki Suk
• Kim Dae Hwan
• Kim Hang Kee
• Kim Hye Mi
• Kim Ki Hong
• Park Su Ran

Classifications

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title

• G06F18/2411

  based on the proximity to a decision surface, e.g. support vector machines · CPC title

• G06T2207/30196

  Human being; Person · CPC title