Image display apparatus using foveated rendering

Patent metadata

Field	Value
Publication number	US-10586392-B2
Application number	US-201715819665-A
Country	US
Kind code	B2
Filing date	Nov 21, 2017
Priority date	Nov 28, 2016
Publication date	Mar 10, 2020
Grant date	Mar 10, 2020

Abstract

Information representing a position and orientation of a captured image or information for deriving the position and orientation is acquired from the captured image as extraction information, and a reduced-information-amount image is generated by reducing an amount of information in the captured image in accordance with a distance from a position that a user is gazing at in the captured image. The reduced-information-amount image and the extraction information are outputted to an external device, and a composite image that has been generated based on the reduced-information-amount image and an image of a virtual space generated by the external device based on the extraction information is received.

First claim

What is claimed is:

1. An image display apparatus, comprising:
one or more processors; and
a memory coupled to the one or more processors, the memory having stored thereon instructions which, when executed by the one or more processors, cause the image display apparatus to:
acquire, from a captured image, information representing a position and orientation of the captured image or information for deriving the position and orientation, as extraction information;
generate, after acquiring the extraction information from the captured image, a reduced-information-amount image by reducing, so that total amount of information for the reduced-information-amount image and the extraction information is smaller than an amount of information in the captured image, the amount of the information in the captured image from which the extraction information was acquired, in accordance with a lateral distance from a position where a user is gazing at in the captured image from which the extraction information was acquired;
output the reduced-information-amount image and the extraction information to an external device; and
receive a composite image generated based on the reduced-information-amount image and an image of a virtual space generated by the external device based on the extraction information.

2. The image display apparatus according to claim 1 , wherein in acquiring the extraction information, the instructions, when executed by the one or more processors, cause the image display apparatus to acquire image coordinates and identification information of an index included in the captured image as the extraction information.

3. The image display apparatus according to claim 1 , wherein in acquiring-the information, which represents the position and orientation of the captured image, the instructions, when executed by the one or more processors, causes the image display apparatus to calculate the information representing the position and orientation of the captured image based on an index included in the captured image, as the extraction information.

4. The image display apparatus according to claim 1 , further comprising an image capturing device configured to capture an image of a physical space that includes an index.

5. The image display apparatus according to claim 1 , further comprising:
detector configured to detect a line of sight direction of the user,
wherein the instructions, when executed by the one or more processors, further cause the image display apparatus to:
specify the position that the user is gazing at in the captured image based on the line of sight direction detected by the detector, and
generate the reduced-information-amount image by reducing the amount of information in accordance with a lateral distance from the specified position increasing.

6. The image display apparatus according to claim 1 , wherein the instructions, when executed by the one or more processors, further cause the image display apparatus to:
generate information corresponding to a hand of the user that is in the captured image, and
transmit the reduced-information-amount image, the extraction information, and the generated information to the external device,
wherein the external device generates the composite image by compositing the captured image with a partial region of the image of the virtual space, the partial region being a region other than a region defined by the information corresponding to the hand.

7. The image display apparatus according to claim 1 , wherein the image display apparatus is a head-mounted display.

8. The image display apparatus according to claim 1 , wherein the image display apparatus is a tablet terminal device.

9. The image display apparatus according to claim 1 , further comprising a display device configured to display the received composite image.

10. An image display method, comprising:
acquiring, from a captured image, information representing a position and orientation of the captured image or information for deriving the position and orientation, as extraction information;
generating, after acquiring the extraction information from the captured image, a reduced-information-amount image by reducing, so that total amount of information for the reduced-information-amount image and the extraction information is smaller than an amount of information in the captured image, the amount of the information in the captured image from which the extraction information was acquired, in accordance with a lateral distance from a position where a user is gazing at in the captured image from which the extraction information was acquired;
outputting the reduced-information-amount image and the extraction information to an external device; and
receiving a composite image generated based on the reduced-information-amount image and an image of a virtual space generated by the external device based on the extraction information.

11. A system comprising an image display apparatus and an external device,
wherein the image display apparatus comprises:
one or more first processors; and
a first memory coupled to the one or more first processors, the first memory having stored thereon instructions which, when executed by the one or more first processors, cause the image display apparatus to:
acquire, from a captured image, information representing a position and orientation of the captured image or information for deriving the position and orientation, as extraction information,
generate, after acquiring the extraction information from the captured image, a reduced-information-amount image by reducing, so that total amount of information for the reduced-information-amount image and the extraction information is smaller than an amount of information in the captured image, the amount of the information in the captured image from which the extraction information was acquired, in accordance with a lateral distance from a position where a user is gazing at in the captured image from which the extraction information was acquired,
output the reduced-information-amount image and the extraction information to the external device, and
receive a composite image generated based on the reduced-information-amount image and an image of a virtual space generated by the external device based on the extraction information; and

wherein the external device comprises:
one or more second processors; and
a second memory coupled to the one or more second processors, the second memory having stored thereon instructions which, when executed by the one or more second processors, cause the external device to:
generate an image of a virtual space based on the extraction information,
generate the composite image based on the generated image of the virtual space and the reduced-information-amount image, and
transmit the generated composite image to the image display apparatus.

Assignees

• Canon Kk

Inventors

• Inoguchi Kazutaka
• Hamaguchi Atsushi

Classifications

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2207/30204

  Marker · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title