Eyewear distortion correction

Patent metadata

Field	Value
Publication number	US-11334972-B2
Application number	US-202017039334-A
Country	US
Kind code	B2
Filing date	Sep 30, 2020
Priority date	Sep 30, 2020
Publication date	May 17, 2022
Grant date	May 17, 2022

Abstract

Systems, methods, and non-transitory computer readable mediums including processes to correct for distortion in images introduced by eyewear (i.e., where the facial region surrounding the eye has a boundary that doesn't match the boundary of an uncovered facial region). The correction includes segmenting images to detect eyewear covered facial regions and facial regions not covered by the eyewear and altering the covered facial regions to match the covered facial boundary to the uncovered facial boundary. Alterations include processing using a machine learning model, applying anti-refraction algorithms, scaling the covered facial region to match boundaries of the uncovered facial region, or a combination thereof.

First claim

What is claimed is:

1. A method for addressing facial distortion in images due to eyewear, the method comprising:
obtaining an image including a face of a subject wearing eyewear, the eyewear including a lens defining a coverage area covering a region of the face, the lens distorting the region of the face in the coverage area to produce a covered facial region within the coverage area of the lens, the covered facial region having a covered facial boundary that is not aligned with an uncovered facial boundary of the face outside the coverage area;
segmenting the obtained image to detect the face and the coverage area of the lens;
altering the covered facial region within the detected coverage area of the lens to match the covered facial boundary to the uncovered facial boundary by applying a machine learing model to the detected covered facial region and the detected face to produce the altered covered facial region, and replacing the covered facial region with the altered covered facial region; and
displaying the obtained image with the altered covered facial region.

2. The method of claim 1 , wherein the segmenting further comprises identifying the covered facial region.

3. The method of claim 1 , wherein the obtaining comprises obtaining the image from a camera system of an eyewear device having an augmented reality optical assembly, a field of view of the camera system overlapping a field of view through the augmented reality optical assembly, and wherein the method further comprises:
generating at least one overlay image from the altered covered facial region; and
presenting the at least one overlay image on the augmented reality optical assembly within the detected coverage area of the lens.

4. The method of claim 1 , wherein the image is one of a series of sequential images and the method further comprises, for each subsequent image of the series of sequential images:
segmenting the subsequent image to detect the face and the coverage area of the lens;
altering the covered facial region within the detected coverage area of the lens to match the covered facial boundary to the uncovered facial boundary; and
displaying the subsequent image with the altered covered facial region.

5. The method of claims 1 , wherein the altered covered facial region is undistorted and wherein the machine learning model is trained using a plurality of images with other faces with eyewear including lenses that distort the covered facial boundary and a corresponding plurality of images with the other faces with eyewear that does not distort the covered facial boundary.

6. A system for addressing facial distortion in images due to eyewear, the system comprising:
an image capture device configured to obtain an image including a face of a subject wearing eyewear, the eyewear including a lens defining a coverage area covering a region of the face, the lens distorting the region of the face in the coverage area to produce a covered facial region within the coverage area of the lens, the covered facial region having a covered facial boundary that is not aligned with an uncovered facial boundary of the face outside the coverage area;
a processor coupled to the image capture device, the processor configured to segment the obtained image to detect the face and the coverage area of the lens and to alter the covered facial region within the detected coverage area of the lens to match the covered facial boundary to the uncovered facial boundary by applying a machine learning model to the detected covered facial region and the detected face to produce the altered covered facial region, and replacing the covered facial region with the altered covered facial region; and
a display coupled to the processor, the display configured to display the obtained image with the altered covered facial region.

7. The system of claim 6 , wherein the processor is further configured to segment the obtained image to identify the covered facial region.

8. The system of claim 6 , further comprising:
an eyewear device having a camera system for obtaining the image and an augmented reality optical assembly, a field of view of the camera system overlapping a field of view through the augmented reality optical assembly;
wherein the processor is further configured to generate at least one overlay image from the altered covered facial region and presenting the at least one overlay image on the augmented reality optical assembly within the detected coverage area of the lens.

9. The system of claim 6 , wherein the image is one of a series of sequential images;
wherein the processor is configured to, for each subsequent image of the series of sequential images, segment the subsequent image to detect the face and the coverage area of the lens and alter the covered facial region within the detected coverage area of the lens to match the covered facial boundary to the uncovered facial boundary; and
wherein the display is configured to, for each subsequent image of the series of sequential images, display the subsequent image with the altered covered facial region.

10. The system of claim 6 , wherein the machine learning model trained using a plurality of images with other faces with eyewear including lenses that distort the covered facial boundary and a corresponding plurality of images with the other faces with eyewear that does not distort the covered facial boundary.

11. A non-transitory computer-readable medium for addressing facial distortion in images due to eyewear, the non-transitory computer-readable medium comprising instructions that, when performed by a processor, configure the processor to performed functions, including functions to:
obtain an image including a face of a subject wearing eyewear, the eyewear including a lens defining a coverage area covering a region of the face, the lens distorting the region of the face in the coverage area to produce a covered facial region within the coverage area of the lens, the covered facial region having a covered facial boundary that is not aligned with an uncovered facial boundary of the face outside the coverage area;
segment the obtained image to detect the face and the coverage area of the lens;
alter the covered facial region within the detected coverage area of the lens to match the covered facial boundary to the uncovered facial boundary by applying a machine learning model to the detected covered facial region and the detected face to produce the altered covered facial region, and replacing the covered facial region with the altered covered facial region; and
display the obtained image with the altered covered facial region.

12. The non-transitory computer-readable medium of claim 11 , wherein the processor is further configured to segment the obtained image to identify the covered facial region.

13. The non-transitory computer-readable medium of claim 11 , wherein the instructions are for use with an eyewear device having a camera system for obtaining the image and an augmented reality optical assembly, a field of view of the camera system overlapping a field of view through the augmented reality optical assembly and wherein the processor is further configured to generate at least one overlay image from the altered covered facial region and presenting the at least one overlay image on the augmented reality optical assembly within the detected coverage area of the lens.

14. The non-transitory computer-readable medium of claim 11 , wherein the image is one of a series of sequential images;
wherein the processor is configured to, for each subsequent image of the series of sequential images, segment the subsequent image to de

Assignees

• Zak Eyal
• Melamed Guy
• Snap Inc

Inventors

• Zak Eyal
• Melamed Guy

Classifications

• G06T11/40

  Filling planar surfaces by adding surface attributes, e.g. adding colours or textures · CPC title

• G06T3/4046

  using neural networks · CPC title

• G06T2207/30201

  Face · CPC title

• H04N1/62 Primary

  Retouching, i.e. modification of isolated colours only or in isolated picture areas only · CPC title

• G06V40/161

  Detection; Localisation; Normalisation · CPC title