Image-based detection of surfaces that provide specular reflections and reflection modification

Patent metadata

Field	Value
Publication number	US-11900569-B2
Application number	US-202217957416-A
Country	US
Kind code	B2
Filing date	Sep 30, 2022
Priority date	Aug 26, 2019
Publication date	Feb 13, 2024
Grant date	Feb 13, 2024

Abstract

Various implementations disclosed herein include devices, systems, and methods that detect surfaces and reflections in such surfaces. Some implementations involve providing a CGR environment that includes virtual content that replaces the appearance of a user or the user's device in a mirror or other surface providing a reflection. For example, a CGR environment may be modified to include a reflection of the user that does not include the device that the user is holding or wearing. In another example, the CGR environment is modified so that virtual content, such as a newer version of the electronic device or a virtual wand, replaces the electronic device in the reflection. In another example, the CGR environment is modified so that virtual content, such as a user avatar, replaces the user in the reflection.

First claim

What is claimed is:

1. A method comprising:
at a processor:
obtaining a first image of a physical environment, the first image comprising a specular reflection of a portion of an object;
obtaining a second image comprising a direct view of the portion of the object;
detecting the specular reflection in the first image based on matching the specular reflection of the portion of the object in the first image with the direct view of the portion of the object in the second image; and
determining a location of a surface in the physical environment that provided the specular reflection based on the specular reflection.

2. The method of claim 1 , wherein the first image and the second image are captured by different image capture devices at approximately a same time.

3. The method of claim 1 , wherein the first image and the second image are portions of a single image captured by an image capture device.

4. The method of claim 1 , wherein the object is a person, clothing, or an electronic device.

5. The method of claim 1 , wherein the matching comprises matching a shape, a size, a color, or a texture in the specular reflection and direct view of the portion of the object.

6. The method of claim 1 , wherein the matching comprises matching movement in:
a first set of images captured over time and comprising specular reflections of the portion of the object; and
a second set of images captured over time and comprising direct views of the portion of the object.

7. A system comprising:
a processor and a computer-readable storage medium comprising instructions that upon execution by the processor cause the system to perform operations, the operations comprising:
obtaining a first image of a physical environment, the first image comprising a specular reflection of a portion of an object;
obtaining a second image of the physical environment, the second image comprising a direct view of the portion of the object;
detecting the specular reflection in the first image based on matching the specular reflection of the portion of the object in the first image with the direct view of the portion of the object in the second image; and
determining a location of a surface in the physical environment that provided the specular reflection based on the specular reflection.

8. A method comprising:
at a processor:
producing an identifier via a light source at a device;
obtaining an image of a physical environment;
determining that the image comprises a specular reflection of the device based on detecting content corresponding to the identifier in the image and determining that the identifier uniquely corresponds to the device; and
determining a location of a surface in the physical environment that provided the specular reflection, the location determined based on the specular reflection in the image.

9. The method of claim 8 , wherein producing the identifier comprises displaying the identifier on a display of the device.

10. The method of claim 8 , wherein determining that the image comprises the specular reflection comprises detecting a display of the device in the image.

11. The method of claim 8 , wherein producing the identifier comprises producing a light pattern via a light source at the device.

12. The method of claim 8 , wherein producing the identifier comprises emitting infrared light from an infrared light source, wherein determining that the image comprises the specular reflection of the device is based on identifying a temporal signature in the infrared light.

13. The method of claim 8 , wherein the identifier changes over time.

14. The method of claim 8 , wherein producing the identifier comprises:
detecting a brightness of the environment; and
selecting a brightness of the identifier based on the brightness of the environment.

15. The method of claim 8 , wherein producing the identifier comprises:
detecting a capability of the device; and
selecting display parameter for displaying the identifier based on the capability.

16. The method of claim 8 , wherein the identifier is produced in synchronization with a camera that obtains the image.

17. The method of claim 8 , wherein determining that the image comprises the specular reflection comprises determining that content in the image does not correspond to a direct view or reflection of another device.

18. The method of claim 17 , wherein determining that the content in the image does not correspond to the direct view or reflection of another device comprises determining that the content in the image has a position and orientation consistent with a reflected device.

19. The method of claim 18 , wherein determining that the content in the image has the position and orientation consistent with the reflected device comprises identifying a symmetry transform.

20. The method of claim 8 further comprising providing a notification based on determining the location of the surface.

Assignees

• Apple Inc

Inventors

• Meier Peter
• Kurz Daniel
• Clark Brian Chris
• Ben Himane Mohamed Selim

Classifications

• G06T15/10

  Geometric effects · CPC title

• G06T5/005 Primary

  Physics · mapped topic

• G06T7/75

  involving models · CPC title

• G06V10/60

  relating to illumination properties, e.g. using a reflectance or lighting model · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title