Perspective-correct passthrough architectures for head-mounted displays

Patent metadata

Field	Value
Publication number	US-2024251176-A1
Application number	US-202418420349-A
Country	US
Kind code	A1
Filing date	Jan 23, 2024
Priority date	Jan 25, 2023
Publication date	Jul 25, 2024
Grant date	—

Abstract

According to examples, an image capture device may include a lens array including lenses supported by a lens support structure, in which the lenses are arranged to capture light rays from multiple view-points. The image capture device may also include a sensor to capture light and convert the captured light into data that is used to form an image and a plurality of apertures positioned between the lenses and the sensor, in which the apertures are positioned with respect to the lenses and the sensor to allow rays of light that would have reached a certain reference location spaced from the sensor to pass through the plurality of apertures. An optimization algorithm may be applied to the collected sensor data to reproduce an image of a scene at a perspective that is intended to match a virtual eye position that is offset from the location of the sensor.

First claim

1 . An image capture device, comprising:
a lens array including a plurality of lenses supported by a lens support structure, wherein the plurality of lenses are arranged to capture light rays from multiple view-points; a sensor to capture light and convert the captured light into data that is used to form an image; and a plurality of apertures positioned between the lenses and the sensor, wherein the plurality of apertures are positioned with respect to the lenses and the sensor to allow rays of light that would have reached a certain reference location spaced from the sensor to pass through the plurality of apertures.

2 . The image capture device of claim 1 , wherein sections between the plurality of apertures are positioned to physically block rays of light that would not have reached the certain reference location through the plurality of lenses.

3 . The image capture device of claim 1 , wherein the plurality of lenses spatially multiplex a spatio-angular light field impinging on the sensor onto different regions of the sensor.

4 . The image capture device of claim 1 , wherein the plurality of lenses selectively pick off incoming light rays that converge to the certain reference location.

5 . The image capture device of claim 1 , wherein each of the plurality of lenses has a respective pupil that only accepts rays within a predefined angular range for a corresponding position on the sensor.

6 . The image capture device of claim 5 , wherein prescriptions, sizes, and locations of the pupils are jointly optimized for a geometry of the sensor and the certain reference location.

7 . The image capture device of claim 1 , wherein the certain reference location comprises a target center of perspective of a user of the image capture device.

8 . The image capture device of claim 1 , wherein the image capture device is to be mounted on a front side of a head-mountable display to capture images of an environment in front of the head-mountable display and wherein the certain reference location comprises a virtual eye position behind the head-mountable display.

9 . A head-mounted display, comprising:
a chassis having a front side and a back side; and an image capture device mounted to the front side of the chassis, the image capture device including:
a lens array including a plurality of lenses supported by a lens support structure, wherein the plurality of lenses are arranged to capture light rays from multiple view-points;
a sensor to capture light and convert the captured light into data that is used to form an image; and
a plurality of apertures positioned between the lenses and the sensor, wherein the plurality of apertures are positioned with respect to the lenses and the sensor to allow rays of light that would have reached a virtual eye position spaced from the sensor to pass through the plurality of apertures, wherein the virtual eye position is positioned behind the back side of the chassis.

10 . The head-mounted display of claim 9 , further comprising:
a processor; and a memory on which is stored machine-readable instructions that when executed by the processor, cause the processor to:
access raw sensor data having a plurality of sub-aperture views of a scene, wherein the plurality of sub-aperture views comprise views of light that would have reached the virtual eye position that is spaced from a sensor that captured the raw sensor data;
apply a reconstruction algorithm on the plurality of sub-aperture views; and
apply gradient domain image stitching to the plurality of sub-apertures following the application of the reconstruction algorithm to generate a stitched image, wherein the stitched image is to accurately reproduce the image at a perspective that matches the virtual eye position.

11 . The head-mounted display of claim 10 , wherein the raw sensor data was captured by the image capture device, the head-mounted display having a display positioned on the back side of the chassis, the instructions further causing the processor to:
cause the stitched image to be displayed on the display of the head-mounted display.

12 . The head-mounted display of claim 9 , wherein the lens support structure is to physically block rays of light that would not have reached the virtual eye position through the plurality of lenses.

13 . The head-mounted display of claim 9 , wherein the plurality of lenses spatially multiplex a spatio-angular light field impinging on the sensor onto different regions of the sensor.

14 . The head-mounted display of claim 9 , wherein the plurality of lenses selectively pick off incoming light rays that converge to the virtual eye position.

15 . The head-mounted display of claim 9 , wherein each of the plurality of lenses has a respective pupil that only accepts rays within a predefined angular range for a corresponding position on the sensor.

16 . The head-mounted display of claim 15 , wherein prescriptions, sizes, and locations of the pupils are jointly optimized for a geometry of the sensor and the virtual eye position.

17 . The head-mounted display of claim 9 , wherein the virtual eye position comprises a target center of perspective of a user of the head-mounted display.

18 . A method comprising:
accessing, by a processor, raw sensor data having a plurality of sub-aperture views of a scene, wherein the plurality of sub-aperture views comprise views of light that would have reached a virtual eye position that is spaced from a sensor that captured the raw sensor data; applying, by the processor, a reconstruction algorithm on the plurality of sub-aperture views; and applying, by the processor, gradient domain image stitching to the plurality of sub-apertures following application of the reconstruction algorithm to generate a stitched image, wherein the stitched image is to accurately reproduce the image at a perspective that matches the virtual eye position.

19 . The method of claim 18 , wherein the raw sensor data was captured by an image capture device positioned on a front side of a head-mounted display, the head-mounted display having a display positioned on a back side of the head-mounted display, the method further comprising:
causing the stitched image to be displayed on the display of the head-mounted display.

20 . The method of claim 19 , wherein the image capture device comprises:
a lens array including a plurality of lenses supported by a lens support structure, wherein the plurality of lenses are arranged to capture light rays from multiple view-points; a sensor to capture light and convert the captured light into data that is used to form an image; and a plurality of apertures positioned between the lenses and the sensor, wherein the plurality of apertures are positioned with respect to the lenses and the sensor to allow rays of light that would have reached a virtual eye position spaced from the sensor to pass through the plurality of apertures, wherein the virtual eye position is positioned behind the back side of the image capture device.

Assignees

• Meta Platforms Tech Llc

Inventors

• Matsuda Nathan
• Kuo Grace Elizabeth
• Lanman Douglas Robert
• Penner Eric
• Smith Clinton
• Moczydlowski Seth
• Ching Alexander

Classifications

• G06V40/193

  Preprocessing; Feature extraction · CPC title

• G06V10/16

  using multiple overlapping images; Image stitching · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• H04N23/957 Primary

  Light-field or plenoptic cameras or camera modules · CPC title