Image processing system, image processing apparatus, storage medium having stored therein image processing program, and image processing method

Patent metadata

Field	Value
Publication number	US-9324298-B2
Application number	US-201414283502-A
Country	US
Kind code	B2
Filing date	May 21, 2014
Priority date	Jun 13, 2013
Publication date	Apr 26, 2016
Grant date	Apr 26, 2016

Abstract

A marker image that, if included in a captured image captured by an image capturing apparatus, performs image processing on the captured image and is thereby allowed to cause a predetermined virtual object to appear is placed in a virtual space. Then, a viewpoint for displaying a part of the virtual space on a display apparatus is changed in accordance with a user input, and the virtual space viewed from the viewpoint is displayed on the display apparatus.

First claim

What is claimed is:

1. An image processing system including a first information processing apparatus for causing an image of a predetermined virtual space to be displayed on a first display apparatus, and a second information processing apparatus for causing an image of real space captured by an image capturing apparatus to be displayed on a second display apparatus,
the first information processing apparatus comprising:
one or more processors configured to:
acquire a user input;
place in the virtual space a marker image that is allowed to cause a corresponding predetermined virtual object to appear;
in accordance with the user input, change a viewpoint for displaying a part of the virtual space on the first display apparatus; and
display on the first display apparatus the virtual space viewed from the changed viewpoint,

the second information processing apparatus comprising:
one or more processors configured to:
acquire a captured image captured by the image capturing apparatus;
if the marker image is included in the captured image, combine the virtual object corresponding to the marker image with the captured image at a position corresponding to the marker image in the captured image; and
if the virtual object has been combined, display on the second display apparatus the captured image with which the virtual object has been combined.

2. The image processing system according to claim 1 , wherein
in accordance with the user input, a position of the viewpoint is allowed to be changed so that a distance from the viewpoint to the marker image in the virtual space changes.

3. The image processing system according to claim 1 , wherein
the virtual space includes a plurality of panoramic images associated with spots on a map,
the marker image is placed at a predetermined position of at least one panoramic image associated with at least one of the spots,
one of the spots on the map is selected as a current spot based on the user input,
and the panoramic image associated with the current spot is selected from the plurality of panoramic images, thereby changing the viewpoint, and
in the display on the first display apparatus, an image of, in the panoramic image associated with the current spot, a partial area determined in accordance with a user operation is displayed on the first display apparatus.

4. The image processing system according to claim 3 , wherein
in the display on the first display apparatus, if the marker image is placed at another spot that satisfies a predetermined condition with respect to the current spot, another spot marker image is displayed together with the panoramic image associated with the current spot on the first display apparatus, the other spot marker image representing the marker image placed at the other spot when viewed from a viewpoint of the panoramic image at the current spot.

5. The image processing system according to claim 4 , wherein
the one or more processors of the first information processing apparatus are further configured to, based on a direction from the current spot to the other spot on the map, determine a placement position of the other spot marker image in the panoramic image associated with the current spot.

6. The image processing system according to claim 5 , wherein
based on a positional relationship on the map between a viewpoint at the other spot and the marker image placed at the other spot, the placement position of the other spot marker image is determined in the panoramic image associated with the current spot.

7. The image processing system according to claim 6 , wherein
based on a distance from the current spot to the other spot calculated based on information indicating a position of a spot on the map, the placement position of the other spot marker image is determined in the panoramic image associated with the current spot.

8. The image processing system according to claim 5 , wherein
a direction of the other spot marker image when viewed from the viewpoint at the current spot is determined so as to represent a direction of the marker image placed at the other spot when viewed from the viewpoint at the current spot.

9. The image processing system according to claim 4 , wherein
it is determined whether or not the marker image is placed at another spot adjacent to the current spot on the map and/or another spot within a predetermined range from the current spot on the map, and if the marker image is placed, the other spot marker image is displayed together with the panoramic image associated with the current spot on the first display apparatus.

10. The image processing system according to claim 3 , wherein
the panoramic images are generated by panoramically photographing the plurality of spots in real space, and
a virtual image having a predetermined design is placed on the panoramically photographed panoramic images in a superimposing manner, thereby placing the marker image.

11. The image processing system according to claim 1 , wherein
the marker image is placed in the virtual space in accordance with the user input.

12. An image processing apparatus for causing an image of a predetermined virtual space to be displayed on a display apparatus, the image processing apparatus comprising:
one or more processors configured to:
acquire a user input;
place in the virtual space a marker image that is captured by an image capturing apparatus and is thereby allowed to cause a corresponding predetermined virtual object to appear in a captured image;
in accordance with the user input, change a viewpoint for displaying a part of the virtual space on the display apparatus; and
display on the display apparatus the virtual space viewed from the changed viewpoint.

13. A non-transitory computer-readable storage medium having stored therein
an image processing program to be executed by a computer included in an information processing apparatus for causing an image of a predetermined virtual space to be displayed on a display apparatus, the image processing program causing the computer to execute:
acquiring a user input;
placing in the virtual space a marker image that is captured by an image capturing apparatus and is thereby allowed to cause a corresponding predetermined virtual object to appear in a captured image;
in accordance with the user input, changing a viewpoint for displaying a part of the virtual space on the display apparatus; and
displaying on the display apparatus the virtual space viewed from the changed viewpoint.

14. An image processing method to be executed by a processor or a cooperation of a plurality of processors, the processor or the plurality of processors contained in an image processing system including a first information processing apparatus for causing an image of a predetermined virtual space to be displayed on a first display apparatus, and a second information processing apparatus for causing an image of real space captured by an image capturing apparatus to be displayed on a second display apparatus, the image processing method comprising:
acquiring a user input;
placing in the virtual space a marker image that is allowed to cause a corresponding predetermined virtual object to appear;
in accordance with the user input, changing a viewpoint for displaying a part of the virtual space on the first display apparatus; and
displaying on the first display apparatus the virtual space viewed from the changed viewpoint,
acquiring a captured image captured by the image capturing apparatus;
if the marker image is included in the captured image, combining the virtual object corresponding to the marker image with

Assignees

• Nintendo Co Ltd

Inventors

• Umehara Akihiro
• Suzuki Toshiaki

Classifications

• G06F3/1438 Primary

  using more than one graphics controller · CPC title

• G09G5/12 Primary

  Synchronisation between the display unit and other units, e.g. other display units, video-disc players · CPC title

• G06T15/20

  Perspective computation · CPC title

• G09G2380/10

  Automotive applications · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title