Automatic image-based skin diagnostics using deep learning

Patent metadata

Field	Value
Publication number	US-11832958-B2
Application number	US-202218080331-A
Country	US
Kind code	B2
Filing date	Dec 13, 2022
Priority date	Dec 4, 2018
Publication date	Dec 5, 2023
Grant date	Dec 5, 2023

Abstract

There is shown and described a deep learning based system and method for skin diagnostics as well as testing metrics that show that such a deep learning based system outperforms human experts on the task of apparent skin diagnostics. Also shown and described is a system and method of monitoring a skin treatment regime using a deep learning based system and method for skin diagnostics.

First claim

What we claim is:

1. A skin diagnostic device comprising:
a memory configured to store and provide a convolutional neural network (CNN) configured to classify pixels of an image to determine a plurality (N) of respective skin sign diagnoses for each of a plurality (N) of respective skin signs wherein the CNN comprises a deep neural network for image classification configured to generate the N respective skin sign diagnoses and wherein the CNN is trained using skin sign data for each of the N respective skin signs; and
at least one processor coupled to the memory and configured to receive the image and process the image using the CNN to generate the N respective skin sign diagnoses,
wherein the CNN comprises:
an encoder phase for image classification and configured to encode features to a final encoder phase feature net; and
a decoder phase configured to receive the final encoder phase feature net for decoding by a plurality (N) of respective parallel skin sign branches to generate each of the N respective skin sign diagnoses.

2. The skin diagnostic device according to claim 1 , wherein the decoder phase includes a global pooling operation to process the final encoder phase feature net to provide to each of the N respective parallel skin sign branches.

3. The skin diagnostic device according to claim 1 , wherein the CNN is further configured to classify the pixels to determine an ethnicity vector and the CNN is trained using skin sign data for each of the N respective skin signs and a plurality of ethnicities.

4. The skin diagnostics device according to claim 3 , wherein the decoder phase comprises a further parallel branch for ethnicity to generate the ethnicity vector.

5. The skin diagnostic device according to claim 1 , wherein each branch of the N respective parallel skin sign branches comprises in succession: a first fully connected layer followed, a first activation layer, a second fully connected layer, a second activation layer and a final activation layer to output a final value comprising one of the N respective skin sign diagnoses and the ethnicity vector.

6. The skin diagnostic device according to claim 5 , wherein the final activation layer is defined in accordance with a function of equation (1) for an input score x received from the second activation layer:

LeakyClamp
⁡
(
x
)

=

{

x

if
⁢

x

∈

[

a
,
b

]

α
⁡
(

x
-
a

)

+
a

if
⁢

x

<
a

α
⁡
(

x
-
b

)

+
b

if
⁢

x

>
b

(
1
)

where α is a slope, a is a lower bound and b is an upper bound of a respective score range for each the N respective skin sign diagnoses.

7. The skin diagnostic device according to claim 4 :
wherein the CNN is trained using multiple samples in the form (x i , y i ), with x i being the i-th training image and y i being a corresponding vector of ground truth skin sign diagnoses; and
wherein the CNN is trained to minimize a loss function for each respective branch of the N parallel skin sign branches and the further parallel branch for ethnicity.

8. The skin diagnostic device according to claim 7 , wherein the CNN is further trained to minimize a loss function L, comprising a L2 loss function for each of the N respective skin sign branches in a weighted combination with a standard cross-entropy classification loss L ethnicity for the further parallel branch for ethnicity, according to equation (3):
L=L 2+λ L ethnicity   (3)

where λ controls a balance between a score regression and ethnicity classification losses.

9. The skin diagnostic device according to claim 1 , wherein the memory stores a face and landmark detector to pre-process the image and wherein the at least one processor is configured to generate a normalized image from the image using the face and landmark detector and use the normalized image when using the CNN.

10. The skin diagnostic device according to claim 1 , wherein the CNN comprises a pre-trained network for image classification which is adapted to generate the N respective skin sign diagnoses such that:
the fully connected layers of the pre-trained network are removed; and
N respective groups of layers are defined to decode a same feature net for each of the N respective skin sign diagnoses in parallel.

11. The skin diagnostic device according to claim 1 , configured as one of:
a computing device for personal use comprising a mobile dev

Assignees

• Oreal

Inventors

• Jiang Ruowei
• Ma Junwei
• Ma He
• Elmoznino Eric
• Kezele Irina
• Levinshtein Alex
• Despois Julien
• Perrot Matthieu
• Flament Frederic Antoinin Raymond Serge
• Aarabi Parham

Classifications

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/096

  Transfer learning · CPC title

• G06N3/09

  Supervised learning · CPC title