Computer-readable storage medium, image display apparatus, image display system, and image display method

Patent metadata

Field	Value
Publication number	US-10015473-B2
Application number	US-201113158939-A
Country	US
Kind code	B2
Filing date	Jun 13, 2011
Priority date	Jun 11, 2010
Publication date	Jul 3, 2018
Grant date	Jul 3, 2018

Abstract

First, the interval between a left virtual camera and a right virtual camera is determined based on the relative position and the relative orientation of an outer imaging section (left) with respect to a marker, which are calculated from a result of marker recognition in the left real world image, and on the relative position and the relative orientation of an outer imaging section (right) with respect to the marker, which are calculated from a result of marker recognition in the right real world image. After the interval between the left virtual camera and the right virtual camera is determined, the positions and the orientations of the left virtual camera and the right virtual camera are determined based on the determined interval such that the relationship between the position and the orientation of the left virtual camera, and the position and the orientation of the right virtual camera is ideal.

First claim

What is claimed is:

1. A non-transitory computer-readable storage medium having stored therein an image display program which is configured to stereoscopically display a real world image onto which a 3-dimensional virtual object is superimposed on a screen of a stereoscopic display apparatus configured to provide a stereoscopic view to a user, the stereoscopic display apparatus including a computing system that includes at least one processor circuit coupled to a memory device, the memory device configured to store real world image data output by a left camera and a right camera, the image display program comprising instructions that are, when executed by a computing system, configured to:
recognize a predetermined shooting target from the stored real world image data output by the left camera and the right camera;
calculate multiple first position orientation information, which are each based on separate relate world image data acquired over a period of time, indicating relative positions and relative orientations of the right camera and the predetermined shooting target with respect to each other;
calculate multiple second position orientation information, which are each based on separate relate world image data acquired over the period of time, indicating relative positions and relative orientations of the left camera and the predetermined shooting target with respect to each other;
determine how much the calculated multiple first position orientation information and the calculated multiple second position orientation information vary;
as a result of determination that the calculated multiple first position orientation information and the calculated multiple second position orientation information vary less than the predetermined amount, calculate pose data, which is indicative of a single position and a single orientation in a virtual space, as a function of multiple calculated first position orientation information and multiple calculated second position orientation information;
calculate a midpoint, as the single position, between the relative position of the right camera and the relative position of the left camera;
calculate a position and an orientation of a right virtual camera for generating a right virtual space image of the virtual space based on the calculated pose data;
calculate a position and an orientation of a left virtual camera for generating a left virtual space image in the virtual space based on the calculated pose data;
generate the right virtual space image as viewed from the right virtual camera in accordance with the calculated position and the calculated orientation of the right virtual camera;
generate the left virtual space image as viewed from the left virtual camera in accordance with the calculated position and the calculated orientation of the left virtual camera;
superimpose the right virtual space image onto the real world image data outputted from the right camera;
superimpose the left virtual space image onto the real world image data outputted from the left camera; and
output the superimposed images to the stereoscopic display apparatus for a stereoscopic view,
wherein the positions of the right virtual camera and the left virtual camera are symmetrically set about the calculated midpoint.

2. The non-transitory computer-readable storage medium according to claim 1 , wherein the determined position and determined orientation of the right virtual camera and the left virtual camera are determined such that a relative orientation between the right virtual camera and the left virtual camera is the same as the relative orientation between the right camera and the left camera.

3. The non-transitory computer-readable storage medium according to claim 1 , wherein the orientation of the right virtual camera and the orientation of the left virtual camera are respectively calculated based on, among the single orientation and the single position, only the single orientation.

4. The non-transitory computer-readable storage medium according to claim 1 , wherein the position of the right virtual camera and the position of the left virtual camera are respectively calculated based on, among the single orientation and the single position, only the single position.

5. The non-transitory computer-readable storage medium according to claim 1 , wherein the instructions are further configured to determine a distance between the right virtual camera and the left virtual camera,
wherein the position of the right virtual camera and the position of the left virtual camera are separated by the determined distance.

6. The non-transitory computer-readable storage medium according to claim 5 , wherein the distance between the right virtual camera and the left virtual camera is determined based on a parallax between the respective real world image data output by a left camera and a right camera.

7. The non-transitory computer-readable storage medium according to claim 5 , wherein the distance between the right virtual camera and the left virtual camera is determined by using the calculated second position orientation information and the calculated first position orientation information.

8. The non-transitory computer-readable storage medium according to claim 1 , wherein the single orientation is an average of the first orientation information and the second orientation information.

9. The non-transitory computer-readable storage medium of claim 1 , wherein the pose data is a transformation matrix which indicates the single position and the single orientation.

10. An image display apparatus which stereoscopically displays a real world image onto which a 3-dimensional virtual object is superimposed, the image display apparatus comprising:
a memory device configured to store real world image data output by a left camera and a right camera; and
a processing system that includes at least one processor circuit coupled to the memory device, the processing system configured to:
recognize a target object from the stored respective real world image data output by the left camera and the right camera;
calculate multiple first relative information, which are each based on separate relative real world image data acquired from the right camera over a period of time, indicating relative positions and relative orientations of the right camera and the target object with respect to each other;
calculate multiple second relative information, which are each based on separate relative real world image data acquired from the left camera over a period of time, indicating relative positions and relative orientations of the left camera and the target object with respect to each other;
determine how much the calculated multiple first relative information and the calculated multiple second relative information vary;
as a result of determination that the calculated multiple first relative information and the calculated multiple second relative information vary less than the predetermined amount, calculate pose data, which indicates a single position and a single orientation in a virtual space, as a function of the multiple first relative information and the multiple second relative information;
calculate a midpoint, as the single position, between the relative position of the right camera and the relative position of the left camera;
set a position and an orientation of a right virtual camera in the virtual space as a function of the calculated pose data, which is indicative of the single position and the single orientation;
set a position and an orientation of a left virtual camera in the virtual space as a function of the pose data, which is indicative of the single position and the single orientation;
generate a rig

Assignees

• Ito Yuichiro
• Nintendo Co Ltd

Inventors

• Ito Yuichiro

Classifications

• H04N13/0275

  Electricity · mapped topic

• H04N13/004

  Electricity · mapped topic

• H04N13/0239 Primary

  Electricity · mapped topic

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04N13/239 Primary

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title