SPAD array for intensity image sensing on head-mounted displays

Patent metadata

Field	Value
Publication number	US-12309501-B2
Application number	US-202418627616-A
Country	US
Kind code	B2
Filing date	Apr 5, 2024
Priority date	Nov 30, 2020
Publication date	May 20, 2025
Grant date	May 20, 2025

Abstract

An HMD includes a single photon avalanche diode (SPAD) array comprising a plurality of SPAD pixels. The HMD also includes a display positioned to display images for viewing by an eye of a user. The HMD also includes one or more processors and one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the HMD to perform various acts associated with using the SPAD array to capture an image frame of an environment for display to the user.

First claim

We claim:

1. A method for facilitating image capture of low light environments in a manner that avoids read noise associated with processing images in low light environments, the method comprising:
repeatedly configuring each single photon avalanche diode (SPAD) pixel of a plurality of SPAD pixels of a SPAD array and each SPAD pixel of a second plurality of SPAD pixels of a second SPAD array to trigger an avalanche event in response to detecting a photon over a frame capture time period;
counting a number of avalanche events for each SPAD pixel of the SPAD array and for each SPAD pixel of the second SPAD array;
generating an image frame of an environment based on the number of avalanche events for each SPAD pixel of the SPAD array;
generating a second image frame of the environment based on the number of avalanche events for each SPAD pixel of the second SPAD array;
generating a reprojected image of the environment by reprojecting at least a portion of the image frame to correspond to a perspective of an eye of a user;
generating a reprojected second image of the environment by reprojecting at least a portion of the second image frame to correspond to a perspective of a second eye of the user; and
displaying the reprojected image of the environment and the reprojected second image of the environment on a display positioned to display images for viewing by the eye of the user and the second eye of the user.

2. The method of claim 1 , wherein the display is part of an optically transparent display assembly configured to transmit light from the environment toward the eye and the second eye of the user.

3. The method of claim 1 , wherein the display is part of an optically opaque display assembly configured to substantially prevent light from the environment from reaching the eye and the second eye of the user.

4. The method of claim 1 , further comprising using the SPAD array to consecutively capture image frames of the environment for display to the user at a rate of about 30 Hz or greater.

5. The method of claim 1 , wherein a pixel resolution of the SPAD array matches a pixel resolution of the display.

6. A method for facilitating intensity image capture of low light environments in a manner that avoids read noise associated with processing images in low light environments, the method comprising:
capturing an intensity image of an environment using a single photon avalanche diode (SPAD) array comprising a plurality of SPAD pixels;
obtaining depth information associated with the environment;
generating a reprojected image by reprojecting at least a portion of the intensity image of the environment to correspond to a perspective of an eye of a user, the reprojection being based on the depth information associated with the environment; and
displaying the reprojected image on a display positioned to display images for viewing by the eye of the user.

7. The method of claim 6 , wherein a pixel resolution of the SPAD array matches a pixel resolution of the display.

8. The method of claim 6 , wherein a focus of the environment relative to the SPAD array is selectively adjustable via a variable focus lens.

9. The method of claim 6 , wherein the depth information associated with the environment is based on time of flight computations performed using light emitted from an illuminator and detected by the SPAD array.

10. The method of claim 6 , further comprising capturing a second intensity image of the environment using a second SPAD array comprising a second plurality of SPAD pixels.

11. The method of claim 10 , wherein the SPAD array and the second SPAD array are synchronized in at least one of exposure or readout.

12. The method of claim 10 , wherein the depth information associated with the environment is based on stereo depth computations performed using the intensity image captured by the SPAD array and the second intensity image captured by second SPAD array.

13. The method of claim 6 , wherein the SPAD array and the display are part of a head-mounted display (HMD).

14. A method for facilitating computer vision tasks in low light environments in a manner that avoids read noise associated with processing images in low light environments, the method comprising:
repeatedly configuring each single photon avalanche diode (SPAD) pixel of a first plurality of SPAD pixels of a first SPAD array of a first image sensor and each SPAD pixel of a second plurality of SPAD pixels of a second SPAD array of a second image sensor to trigger an avalanche event in response to detecting a photon over a frame capture time period;
counting or tracking avalanche events for each SPAD pixel of the first SPAD array of the first image sensor and for each SPAD pixel of the second SPAD array of the second image sensor; and
facilitating one or more computer vision tasks based on the counting or tracking of the avalanche events for each SPAD pixel of the first SPAD array of the first image sensor and for each SPAD pixel of the second SPAD array of the second image sensor, wherein the one or more computer vision tasks comprise capturing depth information associated with an environment by performing stereo depth computations using images captured by the first SPAD array of the first image sensor and the second SPAD array of the second image sensor.

15. The method of claim 14 , wherein the one or more computer vision tasks further comprise simultaneous localization and mapping (SLAM).

16. The method of claim 15 , wherein the first image sensor and the second image sensor are part of a head-mounted display (HMD).

17. The method of claim 14 , wherein the one or more computer vision tasks further comprise surface reconstruction.

18. The method of claim 17 , wherein the first image sensor and the second image sensor are part of a head-mounted display (HMD).

19. The method of claim 14 , wherein the one or more computer vision tasks further comprise object tracking.

20. The method of claim 19 , wherein the first image sensor and the second image sensor are part of a head-mounted display (HMD).

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Price Raymond Kirk
• Bleyer Michael
• Edmonds Christopher Douglas

Classifications

• H10F39/191

  Photoconductor image sensors · CPC title

• H10F39/18

  Complementary metal-oxide-semiconductor [CMOS] image sensors; Photodiode array image sensors · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G01S7/4914

  of detector arrays, e.g. charge-transfer gates · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title