Method, apparatus, and recording medium for processing image

Patent metadata

Field	Value
Publication number	US-10672180-B2
Application number	US-201715584274-A
Country	US
Kind code	B2
Filing date	May 2, 2017
Priority date	May 2, 2016
Publication date	Jun 2, 2020
Grant date	Jun 2, 2020

Abstract

A method of processing an image by a device obtaining one or more images including captured images of objects in a target space, generating metadata including information about mapping between the one or more images and a three-dimensional (3D) mesh model used to generate a virtual reality (VR) image of the target space, and transmitting the one or more images and the metadata to a terminal.

First claim

What is claimed is:

1. A method, performed by a terminal, of processing an image, the method comprising:
obtaining information regarding an object of interest;
transmitting the information regarding the object of interest, to a device;
receiving, from the device, a plurality of images including an area of the object of interest captured at the device and metadata regarding the plurality of images, wherein the metadata comprises quality correction information for the plurality of images and weight information for a region which includes an overlapping object which overlaps between two images of the plurality of images, and wherein based on the transmitted information, the area of the object of interest is encoded with a first quality and another area in the plurality of images is encoded with a second quality different from the first quality;
obtaining, from the metadata, at least one parameter regarding the capturing at the device;
determining mapping information between the plurality of images and a three-dimensional (3D) mesh model by performing an operation based on the at least one parameter;
converting an image comprising the overlapping object by blending pixel values corresponding to the region which includes the overlapping object in the plurality of images according to a preset weight value, based on the weight information included in the metadata;
generating a virtual reality (VR) image by rendering the converted image based on the mapping information; and
outputting the generated VR image.

2. The method of claim 1 , wherein the mapping information comprises at least one of an angle at which the plurality of images are captured, a position of a capturing device that captures at least one of the plurality of images, a resolution of the 3D mesh model, and information specifying a type of the 3D mesh model.

3. The method of claim 1 , further comprising:
correcting the plurality of images by correcting pixel values of pixels included in the plurality of images, based on the quality correction information,
wherein the generating of the VR image comprises generating the VR image by rendering the corrected plurality of images based on the mapping information.

4. The method of claim 1 , wherein the terminal is configured to generate the VR image without receiving shape information of the 3D mesh model from the device.

5. The method of claim 1 , wherein at least one image of the plurality of images is cropped to remove redundant image information before being transmitted to the terminal.

6. The method of claim 5 , wherein the plurality of images comprises a first image corresponding to a first view angle and a second image corresponding to a second view angle,
wherein the first view angle overlaps with the second view angle at an overlapping view angle, and
wherein the redundant image information comprises a portion of the second image corresponding to the overlapping view angle.

7. The method of claim 1 , wherein the outputting of the generated VR image comprises displaying the generated VR image on a display included in the terminal.

8. A device for processing an image, the device comprising:
a camera;
a communication interface; and
at least one processor configured to:
control the camera to obtain plurality of images,
generate metadata comprising at least one parameter regarding the plurality of images,
control the communication interface to receive information regarding an object of interest,
determine, from among objects included in the plurality of images, an overlapping object which overlaps between two images of the plurality of images,
convert an image comprising the overlapping object by blending pixel values corresponding to a region which includes the overlapping object in the two images according to a preset value,
based on the received information, encode an area of the object of interest with a first quality and another area in the plurality of images with a second quality different from the first quality, and
control the communication interface to transmit the converted image and the metadata to a terminal based on a result of the encoding,

wherein mapping information between the plurality of images and a three-dimensional (3D) mesh model for generating a virtual reality (VR) image is determined at the terminal according to an operation based on the at least one parameter.

9. The device of claim 8 , wherein the mapping information comprises at least one of an angle at which the plurality of images are captured, a position of a capturing device that captures the plurality of images, a resolution of the 3D mesh model, and information specifying a type of the 3D mesh model.

10. The device of claim 8 , wherein
the metadata comprises at least one of quality correction information for the plurality of images, and weight information for a region which includes an object overlapping between two images of the plurality of images.

11. The device of claim 8 , wherein the at least one processor is further configured to obtain information about noise of the plurality of images, and to correct the plurality of images by correcting pixel values of pixels included in the plurality of images, based on the obtained information about the noise, and the communication interface is further configured to transmit the corrected plurality of images to the terminal, together with the metadata.

12. A terminal for processing an image, the terminal comprising:
a communication interface;
an output interface; and
a controller configured to:
obtain information regarding an object of interest,
control the communication interface to transmit the information regarding the object of interest, to a device,
control the communication interface to receive plurality of images including an area of the object of interest captured at the device and metadata regarding the plurality of images, wherein the metadata comprises quality correction information for the plurality of images and weight information for a region which includes an overlapping object which overlaps between two images of the plurality of images, and wherein based on the transmitted information, the area of the object of interest is encoded with a first quality and another area in the plurality of images is encoded with a second quality different from the first quality,
obtain, from the metadata, at least one parameter regarding the capturing at the device,
determine mapping information between the plurality of images and a three-dimensional (3D) mesh model by performing an operation based on the at least one parameter,
convert an image comprising the overlapping object by blending pixel values corresponding to the region which includes the overlapping object in the two images according to a preset weight value, based on the weight information included in the metadata, to project the converted image using a normalized camera pose, and
generate a virtual reality (VR) image by rendering the projected converted image based on the mapping information; and
control the output interface to output the generated VR image.

13. The terminal of claim 12 , wherein the mapping information comprises at least one of an angle at which the plurality of images are captured, a position of a capturing device that captures at least one of the plurality of images, a resolution of the 3D mesh model, and information specifying a type of the 3D mesh model.

14. The terminal of claim 12 , wherein the controller is further configured to correct the plurality of images by correcting pixel values of pixels included in the plurality of images, based on the quality correction informati

Assignees

• Samsung Electronics Co Ltd

Inventors

• Jeong Jae-Yun
• Kim Do-Wan
• Kim Yong-Gyoo
• Lee Gun-Hee
• Lee Jae-Kyeong
• Lee Jin-Bong
• Choi Dai-Woong
• Choi Hyun-Soo

Classifications

• H04N13/279

  the virtual viewpoint locations being selected by the viewers or determined by tracking · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T15/20 Primary

  Perspective computation · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06T15/10

  Geometric effects · CPC title