Apparatus and method for providing mixed reality content

Patent metadata

Field	Value
Publication number	US-10674077-B2
Application number	US-201715794191-A
Country	US
Kind code	B2
Filing date	Oct 26, 2017
Priority date	Sep 19, 2017
Publication date	Jun 2, 2020
Grant date	Jun 2, 2020

Abstract

A mixed reality content providing apparatus is disclosed. The mixed reality content providing apparatus may recognize an OOI included in a 360-degree VR image to generate metadata of the OOI and may provide a user with mixed reality content where the metadata is overlaid on the 360-degree VR image.

First claim

What is claimed is:

1. A mixed reality content providing method comprising:
obtaining a plurality of images generated by photographing a 360-degree region, device information, and capture information from an image obtainer including a plurality of cameras;
stitching the plurality of images to generate one 360-degree virtual reality (VR) image;
converting the one 360-degree VR image into a 360-degree VR image based on a selected projection;
recognizing a target object of interest (OOI) included in the 360-degree VR image based on the selected projection to obtain target object recognition information;
sensing, by a sensor unit, a state of a real target object corresponding to the OOI to obtain sensing information;
generating metadata, based on the device information, the capture information, the target object recognition information, and the sensing information;
mapping the metadata and the 360-degree VR image based on the selected projection to generate mixed reality content and providing the mixed reality content to an electronic device; and
after the providing, by the electronic device, overlaying a text corresponding to the metadata on the 360-degree VR image based on the selected projection,
wherein the overlaying of the text on the 360-degree VR image based on the selected projection is selectively performed.

2. The mixed reality content providing method of claim 1 , wherein the generating of the metadata comprises:
generating a device identifier (ID) table including a manufacturer name of a camera, number of cameras which have photographed the 360-degree region, camera rig information, and a serial number of each of the cameras, which are extracted from the device information;
generating a capture ID table including global positioning system (GPS) information about the camera, compass information about an image, a time stamp, and a file name of the image, which are extracted from the capture information;
generating a target object ID table including a target object name of the OOI, a type of the selected projection, and a position and a size of the OOI included in the 360-degree VR image based on the selected projection, which are extracted from the target object recognition information;
generating a sensing information ID table including GPS information and state information about the real target object, which are extracted from the sensing information; and
generating the metadata including the device ID table, the capture ID table, the target object ID table, and the sensing information ID table.

3. The mixed reality content providing method of claim 2 , wherein the camera rig information is information representing an arrangement type where the plurality of cameras are arranged.

4. The mixed reality content providing method of claim 2 , wherein
the sensing information ID table further comprises a target object name of the real target object, and
the target object name of the real target object is recognized from the target object name of the OOI.

5. The mixed reality content providing method of claim 2 , wherein the generating of the sensing information ID table comprises:
determining whether the real target object and the OOI are the same target object, based on the GPS information about the real target object, the GPS information about the camera, and the compass information about the image;
when the real target object and the OOI are the same target object, generating the target object name of the OOI as the target object name of the real target object; and
adding the target object name of the real target object into the sensing information ID table.

6. The mixed reality content providing method of claim 1 , wherein the obtaining of the target object recognition information comprises recognizing the OOI by using an image recognition method based on a feature vector extracted from the one 360-degree VR image or an image recognition method based on deep learning.

7. The mixed reality content providing method of claim 1 , wherein the obtaining of the target object recognition information comprises:
modifying a plurality of target object images stored in a database, based on a specific projection;
applying the modified plurality of target object images to a deep network model through supervised learning to learn a parameter of the deep network model; and
recognizing the OOI included in the one 360-degree VR image, based on the parameter.

8. A mixed reality content providing apparatus comprising:
an image obtainer including a plurality of cameras and outputting a plurality of images generated by photographing a 360-degree region with the plurality of cameras, device information about the plurality of cameras, and capture information generated in an operation of capturing the plurality of images;
a stitching unit stitching the plurality of images to generate one 360-degree virtual reality (VR) image, the stitching unit being implemented as at least one processor;
an image converter converting the one 360-degree VR image into a 360-degree VR image based on a selected projection;
an object recognizer outputting target object recognition information obtained by recognizing a target object of interest (OOI) included in the 360-degree VR image based on the selected projection;
a sensor unit sensing a state of a real target object corresponding to the OOI to output sensing information; and
a mixed reality managing server generating metadata, based on the device information, the capture information, the target object recognition information, and the sensing information, mapping the metadata and the 360-degree VR image based on the selected projection to generate mixed reality content, and providing the mixed reality content to an electronic device for displaying the mixed reality content.

9. The mixed reality content providing apparatus of claim 8 , wherein the mixed reality managing server generating the metadata which includes a device identifier (ID) table including a manufacturer name of a camera, number of cameras which have photographed the 360-degree region, camera rig information, and a serial number of each of the cameras, which are extracted from the device information, a capture ID table including global positioning system (GPS) information about the camera, compass information about an image, a time stamp, and a file name of the image, which are extracted from the capture information, a target object ID table including a target object name of the OOI, a type of the selected projection, and a position and a size of the OOI included in the 360-degree VR image based on the selected projection, which are extracted from the target object recognition information, and a sensing information ID table including GPS information and state information about the real target object, which are extracted from the sensing information.

10. The mixed reality content providing apparatus of claim 9 , wherein the sensing information ID table further comprises a target object name of the real target object recognized from the target object name of the OOI.

11. The mixed reality content providing apparatus of claim 9 , wherein the mixed reality managing server determines whether the real target object and the OOI are the same target object, based on the GPS information about the real target object, the GPS information about the camera, and the compass information about the image, and when the real target object and the OOI are the same target object, generates the target object name of the OOI as the target object name of the real target object and adds the target object name of the real target object into the sensing information ID table.

12. The mixed reality c

Assignees

• Electronics & Telecommunications Res Inst

Inventors

• KIM Nack Woo
• Lee Byung Tak
• Lee Sei Hyoung
• Lee Hyun Yong
• Lee Hyung Ok
• Kim Young Sun

Classifications

• G06K9/6273

  Physics · mapped topic

• G06T15/205

  Image-based rendering · CPC title

• H04N5/44504

  Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits · CPC title

• G06K2209/27

  Physics · mapped topic

• G01S19/01

  Satellite radio beacon positioning systems transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO · CPC title