Specular surface mapping

Patent metadata

Field	Value
Publication number	US-12373969-B2
Application number	US-202218145825-A
Country	US
Kind code	B2
Filing date	Dec 22, 2022
Priority date	Jun 30, 2020
Publication date	Jul 29, 2025
Grant date	Jul 29, 2025

Abstract

Methods and apparatus for specular surface mapping in which a camera detects reflections of a light source from a specular surface. The detected light sources may be projected onto a celestial sphere as virtual point sources. True positive observations should be tightly clustered on the celestial sphere; thus, false positives may be identified and removed. Specular surface information may then be determined from clusters of the virtual point sources on the celestial sphere. The clusters of virtual point sources on the celestial sphere may be identified and used to identify a surface as a specular surface. The clusters may also be used to extract other information regarding the specular surface, including but not limited to distance to and extent of the specular surface.

First claim

What is claimed is:

1. A device, comprising:
a light source configured to emit light into an environment;
a camera configured to capture images that contain a plurality of instances of reflected light from the light source that have been reflected from a surface in the environment, wherein the camera is located proximate to the light source on the device; and
a controller comprising one or more processors configured to:
analyze the images captured by the camera to detect the instances of reflected light from the light source in the images;
project the detected instances of reflected light on to a celestial sphere as point sources;
detect a cluster of the point sources on the celestial sphere; and
identify the surface as a specular surface based on the detected cluster of point sources on the celestial sphere.

2. The device as recited in claim 1 , wherein the light source is an array of near-infrared (NIR) light-emitting diodes (LEDs) that project an NIR dot pattern into the environment.

3. The device as recited in claim 2 , wherein the controller is further configured to analyze spatial and temporal distribution of amplitude of the instances of reflected light of dots from the surface captured by the NIR camera to determine one or more characteristics of the surface.

4. The device as recited in claim 1 , wherein the light source is a near-infrared (NIR) flood illuminator.

5. The device as recited in claim 1 , wherein the controller is further configured to filter out false positive light sources captured in the images, where true positive observations of the reflections of the light source are tightly clustered on the celestial sphere, and false positives are randomly distributed on the celestial sphere.

6. The device as recited in claim 1 , wherein the instances of reflected light from the light source are detectable by the camera in a volume projected perpendicular from the specular surface into the environment when a normal to the specular surface is within a field of view of the camera, and wherein observations of the instances of reflected light within the volume fall on a same location on the celestial sphere.

7. The device as recited in claim 6 , wherein the controller is further configured to correlate the point sources in the cluster on the celestial sphere with locations of respective ones of the detected instances of reflected light in a 3D mapping volume of the environment to determine extent of the specular surface in the environment.

8. The device as recited in claim 1 , wherein the celestial sphere is a SLAM (simultaneous localization and mapping) technology model.

9. The device as recited in claim 1 , wherein the device further includes one or more sensors configured to captured information about the environment, wherein the controller is further configured to:
process the information captured by the one or more sensors to generate a 3D mapping of the environment; and
integrate information identifying the specular surface into the 3D mapping of the environment.

10. The device as recited in claim 1 , wherein the device is a smartphone or tablet.

11. The device as recited in claim 1 , wherein the device is a head-mounted device (HMD) of an extended reality (XR) system.

12. A method, comprising:
emitting, by a light source, light into an environment;
capturing, by a camera, images that contain a plurality of instances of reflected light from the light source that have been reflected from a surface in the environment;
performing, by one or more processors:
analyzing the images captured by the camera to detect the instances of reflected light from the light source in the images;
projecting the detected instances of reflected light on to a celestial sphere as point sources;
detecting a cluster of the point sources on the celestial sphere; and
identifying the surface as a specular surface based on the detected cluster of point sources on the celestial sphere.

13. The method as recited in claim 12 , wherein the light source is an array of near-infrared (NIR) light-emitting diodes (LEDs) that project an NIR dot pattern into the environment, the method further comprising analyzing spatial and temporal distribution of amplitude of the instances of reflected light of dots from the surface captured by the NIR camera to determine one or more characteristics of the surface.

14. The method as recited in claim 12 , further comprising identifying and discarding false positive light sources captured in the images, where true positive observations of the instances of reflected light from the light source are tightly clustered on the celestial sphere, and false positives are randomly distributed on the celestial sphere.

15. The method as recited in claim 12 , wherein the instances of reflected light from the light source are detectable by the camera in a volume projected perpendicular from the specular surface into the environment when a normal to the specular surface is within a field of view of the camera, and wherein observations of the instances of reflected light within the volume fall on a same location on the celestial sphere, the method further comprising correlating the point sources in the cluster on the celestial sphere with locations of respective ones of the detected reflections in a 3D mapping volume of the environment to determine extent of the specular surface in the environment.

16. The method as recited in claim 12 , further comprising integrating information identifying the specular surface into a 3D mapping of the environment.

17. The method as recited in claim 12 , wherein the light source, camera, and one or more processors are components of a handheld device or of a head-mounted device (HMD).

18. One or more non-transitory computer-readable storage media storing program instructions that when executed on or across one or more processors cause the one or more processors to:
analyze images captured by a camera to detect a plurality of instances of reflected light from a light source that have been reflected from a surface in an environment in the images;
project the detected instances of reflected light on to a celestial sphere as point sources;
detect a cluster of the point sources on the celestial sphere; and
identify the surface as a specular surface based on the detected cluster of point sources on the celestial sphere.

19. The one or more non-transitory computer-readable storage media as recited in claim 18 , wherein the light source is an array of near-infrared (NIR) light-emitting diodes (LEDs) that project an NIR dot pattern into the environment, further comprising program instructions that when executed on or across one or more processors cause the one or more processors to analyze spatial and temporal distribution of amplitude of the instances of reflected light of dots from the surface captured by the NIR camera to determine one or more characteristics of the surface.

20. The one or more non-transitory computer-readable storage media as recited in claim 18 , further comprising program instructions that when executed on or across one or more processors cause the one or more processors to integrate information identifying the specular surface into a 3D mapping of the environment.

Assignees

• Apple Inc

Inventors

• Pillans Luke A
• Movshovich Aleksandr M
• Soldner Nicholas C
• Kurz Daniel
• Zhang Arthur Y

Classifications

• G01N21/00

  Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light (G01N3/00 - G01N19/00 take precedence) · CPC title

• H04N5/74

  Projection arrangements for image reproduction, e.g. using eidophor · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• H04N23/21

  from near infrared [NIR] radiation · CPC title