Computer-readable storage medium having image processing program stored therein, image processing apparatus, image processing system, and image processing method

Patent metadata

Field	Value
Publication number	US-9530249-B2
Application number	US-201113053713-A
Country	US
Kind code	B2
Filing date	Mar 22, 2011
Priority date	Sep 24, 2010
Publication date	Dec 27, 2016
Grant date	Dec 27, 2016

Abstract

When an image of a marker existing in a real space is taken by using an outer camera, an image of a plurality of virtual characters which is taken by a virtual camera is displayed on an upper LCD so as to be superimposed on a taken real image of the real space. The virtual characters are located in a marker coordinate system based on the marker, and when a button operation is performed by a user on a game apparatus, the position and the orientation of each virtual character are changed. Then, when a button operation indicating a photographing instruction is provided by the user, an image being displayed is stored in a storage means.

First claim

What is claimed is:

1. A non-transitory computer-readable storage medium having stored therein an image processing program which is executed by a computer of an image processing apparatus which generates an image in which a virtual object image is superimposed on an image taken by a real camera, the image processing program comprising instructions that are, when executed by the computer, configured to:
obtain a real image of an object which is taken by the real camera;
obtain position/orientation information corresponding to a position and an orientation of the real camera in a real space;
set a position and an orientation of a virtual camera in a virtual space on the basis of the obtained position/orientation information;
change at least one of a position and an orientation of at least a part of a virtual object located in the virtual space in accordance with obtained input, wherein the virtual object is among a plurality of virtual objects located in the virtual space;
generate a virtual object image in accordance with the set position and orientation of the virtual camera;
generate a superimposed image in which the generated virtual object image is superimposed on the obtained real image;
generate a cursor that indicates the virtual object for the superimposed image;
output, to a display screen, the superimposed image with the cursor included thereon; and
store the superimposed image to a storage medium as a still photographed image in accordance with a photographing instruction provided by the user, wherein the cursor is not included with the still photograph image stored to the storage medium.

2. The non-transitory storage medium according to claim 1 , wherein the instructions are further configured to:
detect a specific object included in the real image; and
obtain a relative position and a relative orientation of the real camera with respect to the detected specific object.

3. The non-transitory storage medium according to claim 1 , wherein
the image processing apparatus is connectable to a display device,
the instructions are further configured to:
until the photographing instruction is provided, repeatedly generate the superimposed image and output the repeatedly generated the superimposed image to the display device.

4. The non-transitory storage medium according to claim 1 , wherein
the virtual object includes a first part and a second part, and
the instructions are further configured to change an orientation of the first part such that the first part of the virtual object is turned to the virtual camera in accordance with performed input.

5. The non-transitory storage medium according to claim 4 , wherein the instructions are further configured to:
when the orientation of the first part is changed such that the first part is turned to the virtual camera, set the orientation of the first part to an upper limit or a lower limit of a first range if an orientation of the first part based on the second part is out of the first range.

6. The non-transitory storage medium according to claim 4 , wherein the instructions are further configured to repeatedly change the orientation of the first part such that the first part of the virtual object is turned to the virtual camera in accordance with performed input until the photographing instruction is provided.

7. The non-transitory storage medium according to claim 4 , wherein
the real camera is a stereo camera that includes a real camera for the left eye and a real camera for the right eye,
the real image includes a real image for a left eye and a real image for a right eye taken by respective left and right cameras of the stereo camera,
the obtained position/orientation information includes a position and an orientation of each of the real camera for left eye and the real camera for a right eye of the stereo camera,
the virtual camera includes respective left and right virtual cameras that are set in accordance with the position/orientation information,
the orientation of the first part is changed such that the first part of the virtual object is turned to a midpoint between the left virtual camera and the right virtual camera,
the generated virtual object image includes a virtual object image for a left eye and a virtual object image for a right eye that are images the virtual object for which the orientation of the first part has been changed, and
the generated superimposed image includes a superimposed image for a left eye in which the virtual object image for a left eye is superimposed on the real image for a left eye, and a superimposed image for a right eye in which the virtual object image for a right eye is superimposed on the real image for a right eye.

8. The non-transitory storage medium according to claim 1 , wherein the instructions are further configured to move the virtual object in accordance with obtained input.

9. The non-transitory storage medium according to claim 3 , wherein a moving direction of the virtual object on the display device is the same as a direction of an obtained input.

10. The non-transitory storage medium according to claim 8 , wherein the instructions are further configured to correct the position of the virtual object to be within a second range if a position of the virtual object in the virtual space is outside of the second range when the virtual object is moved in accordance with obtained input.

11. The non-transitory storage medium according to claim 1 , wherein the instructions are further configured to change a pose of the virtual object in accordance with user provided input.

12. The non-transitory storage medium according to claim 1 , wherein the instructions are further configured to select one virtual object from among a plurality of virtual objects located in the virtual space as the object for which the position and/or orientation is changed.

13. An image processing apparatus comprising:
at least one real camera; and
a processing system that includes at least one processor, the processing system configured to:
obtain a real image of an object which is taken by the at least one real camera;
obtain pose information of the at least one real camera;
set a virtual camera at a position and/or orientation in a virtual space based one the obtained pose information;
change at least one of a position and orientation of at least a part of a virtual object located in the virtual space in accordance with obtained user input;
generate an image of the virtual object, as viewed from the set virtual camera, for which the at least one of the position and the orientation has been changed;
generate a superimposed image in which the image of the virtual object is superimposed on the obtained real image;
for each generated superimposed image, store the respective superimposed image to volatile memory and correspondingly output, to a display screen, respective superimposed image from the volatile memory;
in response to a received photographing instruction, store a separate still superimposed image to a non-volatile memory storage medium, where the separate still image is based on a superimposed image previously stored to volatile memory.

14. An image processing system comprising:
an input device configured to acquire input;
a memory storage medium that includes volatile memory and non-volatile memory;
at least one real camera that is configured to acquire a real image of an object; and
a processing system that includes at least one processor, the processing system configured to:
obtain information corresponding to a position and/or orientation of the at least one real camera;
set a virtual camera in a v

Assignees

• Kawamoto Kouichi
• Fujiwara Shinya
• Nishimura Yuuki
• Nintendo Co Ltd
• Hal Laboratory Inc

Inventors

• Kawamoto Kouichi
• Fujiwara Shinya
• Nishimura Yuuki

Classifications

• A63F2300/69

  Involving elements of the real world in the game world, e.g. measurement in live races, real video · CPC title

• G06T2207/30204

  Marker · CPC title

• G06T2207/10012

  Stereo images · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04N13/156

  Mixing image signals · CPC title