Image processing device, image processing method and program

Patent metadata

Field	Value
Publication number	US-8933966-B2
Application number	US-87953610-A
Country	US
Kind code	B2
Filing date	Sep 10, 2010
Priority date	Oct 27, 2009
Publication date	Jan 13, 2015
Grant date	Jan 13, 2015

Abstract

There is provided an image processing device, including: a data storage unit storing feature data indicating a feature of appearance of an object; an environment map generating unit for generating an environment map representing a position of one or more objects existing in a real space based on an input image obtained by imaging the real space using an imaging device and the feature data stored in the data storage unit; and an output image generating unit for generating an output image obtained by erasing an erasing target object from the input image based on a position of the erasing target object specified out of objects present in the input image represented in the environment map and a position of the imaging device.

First claim

What is claimed is:

1. An image processing device, comprising:
circuitry configured to
store feature data indicating a feature of appearance of an object,
generate an environment map representing a position of one or more objects existing in a real space based on an input image obtained by imaging the real space using an imaging device and on the stored feature data, and
generate an output image by erasing an erasing target object from the input image based on a position of the erasing target object being one of the one or more objects existing in the real space recorded in the input image and represented in the environment map, and based on a position of the imaging device,

wherein the erasing target object is an object recorded by the imaging device in the imaging of the real space at a time of the imaging of the real space, and
wherein the circuitry is further configured to
estimate a position of a floor surface or a wall surface in the real space based on a position of a point on a surface of an object other than the erasing target object represented by the environment map, and
generate a CG image based on the position of the floor surface or the wall surface in the estimated real space.

2. The image processing device according to claim 1 ,
wherein the circuitry is further configured to dynamically detect the position of the imaging device based on the input image and the feature data.

3. The image processing device according to claim 2 , wherein the circuitry is further configured to:
determine an erased area being an area in the input image corresponding to the erasing target object based on the position of the erasing target object in the environment map and the position of the imaging device, and
generate the output image by superimposing the CG image that is generated, on the input image for at least a part of the erased area.

4. The image processing device according to claim 3 , wherein the circuitry is further configured to generate the CG image further based on a position of the object other than the erasing target object represented by the environment map and the position of the imaging device.

5. The image processing device according to claim 4 , wherein
the feature data includes data indicating, for one or more points on the surface of each object, whether each point is likely to come in contact with the floor surface or the wall surface in the real space,
and the circuitry is further configured to estimate the position of the floor surface or the wall surface in the real space further based on the feature data.

6. The image processing device according to claim 4 , wherein the circuitry is further configured to generate the CG image of a portion included in the erased area and the portion in which the floor surface or the wall surface in the real space should be shown according to an image of a portion in which the floor surface or the wall surface in the real space is estimated to be shown of the input image.

7. The image processing device according to claim 3 ,
wherein the circuitry is further configured to select an object to be virtually displayed in the output image, and
wherein the circuitry is further configured to additionally arrange the selected object in the CG image.

8. The image processing device according to claim 7 , wherein the object to be virtually displayed, corresponds to a specified object after being moved out of the objects present in the input image and the erasing target object corresponds to the specified object before being moved.

9. The image processing device according to claim 1 , wherein the erasing target object is physically present in the real space.

10. The image processing device according to claim 1 , wherein the erasing target object is physically located in the real space between two objects of the one or more objects.

11. An image processing method performed using an image processing device having a memory storing feature data indicating a feature of appearance of an object, the method comprising steps of:
generating an environment map representing a position of one or more objects existing in a real space based on an input image obtained by imaging the real space using an imaging device and on the feature data stored in the memory;
generating an output image obtained by erasing an erasing target object from the input image based on a position in the environment map of the erasing target object being one of the one or more objects existing in the real space recorded in the input image, and based on a position of the imaging device, the erasing target object being an object recorded by the imaging device in the imaging of the real space at a time of the imaging of the real space;
estimating a position of a floor surface or a wall surface in the real space based on a position of a point on a surface of an object other than the erasing target object represented by the environment map; and
generating a CG image based on the position of the floor surface or the wall surface in the estimated real space.

12. A non-transitory computer readable medium having stored thereon a program that when executed by a computer causes the computer to perform a method, the method comprising:
storing feature data indicating a feature of appearance of an object;
generating unit for generating an environment map representing a position of one or more objects existing in a real space based on an input image obtained by imaging the real space using an imaging device and on the feature data stored;
generating an output image obtained by erasing an erasing target object from the input image based on a position of the erasing target object being one of the one or more objects existing in the real space recorded in the input image represented in the environment map, and based on a position of the imaging device, the erasing target object being an object recorded by the imaging device in the imaging of the real space at a time of the imaging of the real space;
estimating a position of a floor surface or a wall surface in the real space based on a position of a point on a surface of an object other than the erasing target object represented by the environment map; and
generating a CG image based on the position of the floor surface or the wall surface in the estimated real space.

Assignees

• Oi Kenichiro
• Guo Jingjing
• Suto Yasuhiro
• Sony Corp

Inventors

• Oi Kenichiro
• Guo Jingjing
• Suto Yasuhiro

Classifications

• G06T7/0044

  Physics · mapped topic

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2207/20164

  Salient point detection; Corner detection · CPC title

• G06T7/74

  involving reference images or patches · CPC title