True size eyewear in real time

Patent metadata

Field	Value
Publication number	US-12387447-B2
Application number	US-202218068333-A
Country	US
Kind code	B2
Filing date	Dec 19, 2022
Priority date	Mar 22, 2021
Publication date	Aug 12, 2025
Grant date	Aug 12, 2025

Abstract

Methods and systems are disclosed for performing operations comprising: receiving an image that includes a depiction of a face of a user; generating a plurality of landmarks of the face based on the received image; removing a set of interfering landmarks from the plurality of landmarks resulting in a remaining set of landmarks of the plurality of landmarks; obtaining a depth map for the face of the user; and computing a real-world scale of the face of the user based on the depth map and the remaining set of landmarks.

First claim

What is claimed is:

1. A method comprising:
removing, from an object depicted in an image, a set of interfering landmarks from a plurality of landmarks resulting in a remaining set of landmarks of the plurality of landmarks;
obtaining a three-dimensional object representation;
matching each of the remaining set of landmarks to the three-dimensional object representation;
determining, for each of the remaining set of landmarks, a visibility parameter based on the three-dimensional object representation, each visibility parameter indicating a degree of visual detectability of each landmark of the plurality of landmarks in the three-dimensional object representation;
determining, for each of the remaining set of landmarks, a stability parameter as a function of movement of the remaining set of landmarks over a plurality of frames of a video comprising the image, the stability parameter indicating how much each of the remaining set of landmarks moves over a threshold number of frames of the plurality of frames and being computed by detecting changes to positions of each of the remaining set of landmarks over the threshold number of frames, landmarks having less amount of movement being assigned a greater stability parameter score than landmarks that are determined to move more than a threshold amount over the threshold number of frames;
sorting the remaining set of landmarks based on both the visibility parameter of each of the remaining set of landmarks and the stability parameter of each of the remaining set of landmarks; and
computing a real-world scale of the object depicted in the image based on landmarks selected from the sorted remaining set of landmarks.

2. The method of claim 1 , the object comprising a face of a user, further comprising:
generating the plurality of landmarks of the face based on the image; and
obtaining a depth map for the face of the user, the real-world scale of the face being computed based on the depth map.

3. The method of claim 1 , wherein the threshold number of frames is determined based on a frame rate of the video, higher frame rates corresponding to a greater threshold number of frames than lower frame rates.

4. The method of claim 3 , further comprising:
tracking movement of the remaining set of landmarks over the plurality of frames;
computing the stability parameter based on a frame rate of the video; and
varying the threshold number of frames based on the frame rate of the video.

5. The method of claim 1 , wherein determining the visibility parameter comprises:
computing how much of each of the remaining set of landmarks matches a corresponding portion of the three-dimensional object representation; and
setting the visibility parameter for each of the remaining set of landmarks proportional to an amount by which each landmark matches the corresponding portion of the three-dimensional object representation.

6. The method of claim 1 , wherein the set of interfering landmarks comprises at least one of a hair region, one or more facial garments, or a face mask.

7. The method of claim 1 , further comprising applying a machine learning model to the image to identify the plurality of landmarks.

8. The method of claim 1 , further comprising:
selecting a threshold number of top landmarks from the remaining set of landmarks based on visibility or stability parameters associated with each of the remaining set of landmarks, wherein the real-world scale of the object is computed based on the selected threshold of top landmarks.

9. The method of claim 8 , wherein the threshold number comprises two top landmarks.

10. The method of claim 8 , further comprising:
selecting eyes and nose landmarks as the top landmarks in response to determining that the eyes and nose landmarks are associated with greater visibility and stability parameters than other landmarks in the remaining set of landmarks.

11. The method of claim 8 , wherein the top landmarks are selected at random from the remaining set of landmarks by:
identifying a collection of landmarks having visibility and stability parameters that exceed a specified value; and
alternating randomly between different subsets of the collection of landmarks during different portions of the video.

12. The method of claim 11 , wherein a first set of top landmarks are selected for a first subset of the frames, and wherein a second set of top landmarks are selected for a second subset of frames.

13. The method of claim 1 , further comprising:
generating an object segmentation mask for the object depicted in the image; and
applying momentum smoothing to a depth map based on the object segmentation mask.

14. The method of claim 1 , further comprising:
updating the plurality of landmarks as each frame of the video depicting the object is received; and
iteratively correcting the real-world scale of the object based on the updated plurality of landmarks by repeating the removing and obtaining operations for the updated plurality of landmarks.

15. The method of claim 1 , further comprising:
obtaining an augmented reality graphical element comprising augmented reality eyewear;
computing fit factors for a plurality of augmented reality eyewear elements based on physical measurements of corresponding glasses;
ranking the plurality of augmented reality eyewear elements based on the computed fit factors;
selecting the augmented reality graphical element based on the ranking;
identifying a nose bridge landmark based on the remaining set of landmarks; and
positioning the augmented reality graphical element within the image on the object based on the nose bridge landmark.

16. The method of claim 15 , further comprising:
detecting a user's finger overlapping the nose bridge landmark;
adjusting a vertical position of the augmented reality graphical element based on movement of the user's finger; and
positioning the nose bridge landmark of the augmented reality graphical element above the nose bridge landmark based on the vertical position.

17. The method of claim 15 , further comprising:
obtaining rigid material properties associated with the augmented reality graphical element;
deforming the augmented reality graphical element based on the rigid material properties; and
adjusting a scale of the augmented reality graphical element based on the computed real-world scale of the object.

18. The method of claim 1 , further comprising:
computing a distance between the remaining set of landmarks;
retrieving a measure of depth for the remaining set of landmarks; and
generating a scaling factor based on the distance and the measured depth that relates a size of the object in the image to a real-world size of the object, wherein a size of an augmented reality graphical element is modified as a function of the scaling factor.

19. A system comprising:
at least one processor; and
a memory component having instructions stored thereon, when executed by the processor, causes the processor to perform operations comprising:
removing, from an object depicted in an image, a set of interfering landmarks from a plurality of landmarks resulting in a remaining set of landmarks of the plurality of landmarks;
obtaining a three-dimensional object representation;
matching each of the remaining set of landmarks to the three-dimensional object representation;
determining, for each of the remaining set of landmarks, a visibility parameter based on the three-dimensional object representation, each visibility parameter indicating a degree of visual detectabil

Assignees

• Snap Inc

Inventors

• Assouline Avihay
• Berger Itamar
• Luo Jean
• Zohar Matan

Classifications

• G06T2219/2021

  Shape modification · CPC title

• G06T2219/2016

  Rotation, translation, scaling · CPC title

• G06T2207/30201

  Face · CPC title

• G06T2207/20124

  Active shape model [ASM] · CPC title

• G06T2207/20081

  Training; Learning · CPC title