Domain specific image quality assessment

Patent metadata

Field	Value
Publication number	US-2024087097-A1
Application number	US-202318371347-A
Country	US
Kind code	A1
Filing date	Sep 21, 2023
Priority date	Dec 4, 2019
Publication date	Mar 14, 2024
Grant date	—

Abstract

In a technique to assess the blurriness of an image, an image of a face is received, the image including a depiction of lips. A processing device determines a region of interest in the image, wherein the region of interest comprises an area inside of the lips. The processing device applies a focus operator to the pixels within the region of interest, and calculates a sharpness metric for the region of interest using an output of the focus operator. The processing device determines whether the sharpness metric satisfies a sharpness criterion, and one or more additional operations are performed responsive to determining that the sharpness metric satisfies the sharpness criterion.

First claim

1 . (canceled)

2 . A computing device comprising:
a memory to store instructions; and a processor operatively coupled to the memory, wherein execution of the instructions causes the processor to:
receive an image comprising a face of a person;
calculate a sharpness metric for a region of interest in the image, wherein data for pixels inside of the region of interest is used to calculate the sharpness metric, and wherein data for pixels of a second region that is outside of the region of interest is not used to calculate the sharpness metric;
determine whether the sharpness metric satisfies a sharpness criterion; and
perform one or more additional operations responsive to determining that the sharpness metric satisfies the sharpness criterion, wherein the one or more additional operations comprise generating a modified version of the image, wherein the region of interest is replaced in the modified version of the image.

3 . The computing device of claim 2 , wherein the image includes a depiction of lips of the person, and wherein the region of interest comprises an area inside of the lips.

4 . The computing device of claim 2 , wherein the processor is further to:
apply a focus operator to pixels of the image that are within the region of interest, wherein the sharpness metric is calculated using an output of the focus operator.

5 . The computing device of claim 4 , wherein applying the focus operator comprises:
applying a Gaussian filter to the pixels within the region of interest; and applying a Laplacian filter to an output of the Gaussian filter.

6 . The computing device of claim 5 , wherein calculating the sharpness metric comprises:
calculating a variance based on an output of the Laplacian filter.

7 . The computing device of claim 4 , wherein applying the focus operator to the pixels within the region of interest comprises applying the focus operator only to the pixels within the region of interest.

8 . The computing device of claim 4 , wherein the processor is further to:
convert pixels within the region of interest to grayscale prior to applying the focus operator to the pixels within the region of interest.

9 . The computing device of claim 2 , wherein the region of interest comprises a depiction of a smile of the person, and wherein the region of interest is replaced with a depiction of a new smile of the person.

10 . The computing device of claim 9 , wherein the region of interest comprises the face of the person.

11 . The computing device of claim 2 , wherein the processor is further to determine the region of interest by processing the image using a trained machine learning model, wherein an output of the trained machine learning model is a probability map that identifies, for each pixel in the image, a probability that the pixel is within the region of interest.

12 . The computing device of claim 2 , wherein the processor is further to determine the region of interest by:
generating a bounding shape around the region of interest using a trained machine learning model; and determining, for each pixel in the image, whether the pixel is inside of the bounding shape or outside of the bounding shape, wherein pixels inside of the bounding shape are associated with the region of interest, and wherein pixels outside of the bounding shape are associated with the second region.

13 . The computing device of claim 2 , wherein the region of interest comprises a current dentition of the person, and wherein the processor is further to:
determine a post-treatment dentition of the person, wherein the current dentition of the person is replaced with the post-treatment dentition of the person in the modified version of the image.

14 . The computing device of claim 2 , wherein the processor is further to:
identify pixels comprising specular highlights in the image; and update at least one of the image or a mask that identifies pixels in the region of interest to remove the specular highlights.

15 . The computing device of claim 2 , wherein the processor is further to:
determine an image class for the image; and determine the sharpness criterion based at least in part on the image class.

16 . The computing device of claim 2 , wherein the processor is further to:
perform pixel intensity normalization on the image.

17 . A non-transitory computer readable medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising:
receiving an image comprising a face of a person; calculating a sharpness metric for a region of interest in the image, wherein data for pixels inside of the region of interest is used to calculate the sharpness metric, and wherein data for pixels of a second region that is outside of the region of interest is not used to calculate the sharpness metric; determining whether the sharpness metric satisfies a sharpness criterion; and performing one or more additional operations responsive to determining that the sharpness metric satisfies the sharpness criterion, wherein the one or more additional operations comprise generating a modified version of the image, wherein the region of interest is replaced in the modified version of the image.

18 . The non-transitory computer readable medium of claim 17 , wherein the image includes a depiction of lips of the person and of a current dentition of the person within the lips, wherein the region of interest comprises an area inside of the lips, and wherein the modified version of the image comprises a post treatment dentition of the person inside of the lips.

19 . The non-transitory computer readable medium of claim 17 , the operations further comprising:
determining the region of interest by processing the image using a trained machine learning model, wherein an output of the trained machine learning model is a pixel-level classification that identifies, for each pixel in the image, an indication as to whether that the pixel is within the region of interest.

20 . A method comprising:
receiving an image comprising a face of a person, the image showing a current dentition of the person; calculating a sharpness metric for a region of interest in the image, wherein the region of interest comprises the current dentition of the person, wherein data for pixels inside of the region of interest is used to calculate the sharpness metric, and wherein data for pixels of a second region that is outside of the region of interest is not used to calculate the sharpness metric; determining whether the sharpness metric satisfies a sharpness criterion; and performing one or more additional operations responsive to determining that the sharpness metric satisfies the sharpness criterion, wherein the one or more additional operations comprise generating a modified version of the image, wherein the current dentition of the person is replaced with a post treatment dentition of the person in the modified version of the image.

21 . The method of claim 20 , further comprising:
determining the region of interest by processing the image using a trained machine learning model, wherein an output of the trained machine learning model is a pixel-level classification that identifies, for each pixel in the image, an indication as to whether that the pixel is within the region of interest.

Assignees

• Align Technology Inc

Inventors

• Shi Chao
• Li Yingjie
• Brown Chad Clayton
• Cramer Christopher E

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06T5/003 Primary

  Physics · mapped topic