Automatic correction of facial sentiment of portrait images

Patent metadata

Field	Value
Publication number	US-10198791-B2
Application number	US-201615236700-A
Country	US
Kind code	B2
Filing date	Aug 15, 2016
Priority date	Aug 15, 2016
Publication date	Feb 5, 2019
Grant date	Feb 5, 2019

Abstract

Techniques are disclosed for correcting facial sentiment of digital images. Facial data captured in a target image is analyzed to obtain facial-based sentiment. A favored sentiment is determined based at least in part on the facial-based sentiment. The favored sentiment is then applied to at least one face included in the target image that doesn't reflect the favored sentiment. Analyzing facial data may include detecting facial landmarks that are good indicators of sentiment (e.g., eyes, mouth, eyebrows, jawline, and nose). Such landmarks can be processed, with supervised machine learning, to detect the corresponding facial sentiment. A favored sentiment of the target image is thus identified, and can be applied to one or more non-compliant faces in the target image. In some embodiments, the favored sentiment can be further based on a plurality of additional sentiment indicators, including geo data, text, and/or other images associated with the target image.

First claim

What is claimed is:

1. A computer-implemented method for automatically correcting facial sentiment of a digital image, the method comprising:
receiving a request to correct facial sentiment in a given target image;
analyzing facial data of the target image to obtain facial-based sentiment;
determining a winning sentiment based on the facial-based sentiment; and
applying the winning sentiment to at least one face included in the target image, wherein applying the winning sentiment comprises
identifying faces in the target image that do not match the winning sentiment; and
modifying one or more landmark features of each face that does not match the winning sentiment, thereby producing an edited version of the target image.

2. The method of claim 1 , wherein analyzing facial data of the target image to obtain the facial-based sentiment comprises:
detecting a face in the target image;
detecting landmarks of the detected face;
processing, with supervised machine learning and classification, the landmarks of the detected face to obtain the facial-based sentiment; and
identifying and outputting the winning sentiment.

3. The method of claim 2 , wherein the supervised machine learning and classification used to process the landmarks of the detected face to obtain the facial-based sentiment is carried out using a support vector machine (SVM).

4. The method of claim 1 , further comprising at least one of:
analyzing other images related to the target image to obtain supplemental facial-based sentiment, wherein the other images are related to the target image based on a time and location at which they were captured;
analyzing geo data of the target image to obtain geo-based sentiment; and
analyzing textual data associated with the target image to obtain textual-based sentiment;
wherein the winning sentiment is further based on at least one of the textual-based sentiment, the supplemental facial-based sentiment, and the geo-based sentiment.

5. The method of claim 4 , wherein analyzing other images related to the target image to obtain the supplemental facial-based sentiment comprises:
identifying one or more additional images related to the target image;
detecting a face in the additional images;
detecting landmarks of the detected face;
processing, with supervised machine learning and classification, the landmarks of the detected face to obtain the supplemental facial-based sentiment;
tracking supplemental facial-based facial sentiment of a plurality of people captured in the additional images; and
identifying and outputting a most-favored facial sentiment of the additional images.

6. The method of claim 4 , wherein analyzing geo data of the target image to obtain the geo-based sentiment comprises:
detecting a geographic location at which the target image was captured; and
identifying and outputting the geo-based a-sentiment of the target image based on the geographic location.

7. The method of claim 4 , wherein analyzing textual data associated with the target image to obtain the textual-based sentiment comprises:
detecting textual content associated with the target image, wherein the textual content is at least one of within and proximate to the target image;
determining sentiment of the detected textual content; and
identifying and outputting the textual-based sentiment of the target image based on the determined.

8. A computer program product including one or more non-transitory machine readable mediums encoded with instructions that when executed by one or more processors cause a process to be carried out for automatically correcting facial sentiment of a digital image, the process comprising:
receiving a request to correct facial sentiment in a given target image;
analyzing facial data of the target image to obtain facial-based sentiment;
determining a winning sentiment based on the facial-based sentiment; and
applying the winning sentiment to at least one face included in the target image, wherein applying the winning sentiment comprises
identifying faces in the target image that do not match the winning sentiment; and
modifying one or more landmark features of each face that does not match the winning sentiment, thereby producing an edited version of the target image.

9. The computer program product of claim 8 , wherein analyzing facial data of the target image to obtain the facial-based sentiment comprises:
detecting a face in the target image;
detecting landmarks of the detected face;
processing, with supervised machine learning and classification, the landmarks of the detected face to obtain the facial-based sentiment; and
identifying and outputting the winning sentiment.

10. The computer program product of claim 9 , the process further comprising at least one of:
analyzing other images related to the target image to obtain supplemental facial-based sentiment;
analyzing geo data of the target image to obtain geo-based sentiment; and
analyzing textual data associated with the target image to obtain textual-based sentiment;
wherein the winning sentiment is further based on at least one of the textual-based sentiment, the supplemental facial-based sentiment, and the geo-based sentiment.

11. The computer program product of claim 10 , wherein the other images are related to the target image based on a time and location at which they were captured, and wherein analyzing the other images related to the target image to obtain the supplemental facial-based sentiment comprises:
identifying one or more additional images related to the target image;
detecting a face in the additional images;
detecting landmarks of the detected face;
processing, with supervised machine learning and classification, the landmarks of the detected face to obtain the supplemental facial-based sentiment;
tracking supplemental facial-based sentiment of a plurality of people captured in the additional images; and
identifying and outputting a most-favored facial sentiment of the additional images.

12. The computer program product of claim 10 , wherein analyzing geo data of the target image to obtain the geo-based sentiment comprises:
detecting a geographic location at which the target image was captured; and
identifying and outputting the geo-based a-sentiment of the target image based on the geographic location.

13. The computer program product of claim 10 , wherein analyzing textual data associated with the target image to obtain the textual-based sentiment comprises:
detecting textual content associated with the target image, wherein the textual content is at least one of within and proximate to the target image;
determining sentiment of the detected textual content; and
identifying and outputting the textual-based sentiment of the target image based on the determined sentiment.

14. A system for automatically correcting facial sentiment of a digital image, comprising:
an input/request circuit to receive a request to correct facial sentiment in a given target image;
a facial data analysis circuit to analyze facial data of the target image to obtain facial-based sentiment;
a voting circuit to determine a winning sentiment based on the facial-based sentiment; and
a face adjust circuit to apply the winning sentiment to at least one face included in the target image, wherein the face adjust circuit applies the winning sentiment to at least one face included in the target image by
identifying faces in the target image that do not match the winning sentiment, and
modifying one or more landmark features of each face that does not match the winning sentiment, thereby producing an edited versio

Assignees

• Adobe Inc

Inventors

• Angara Srinivasa Madhava Phaneendra
• Lewis Lance
• Dhawan Anmol

Classifications

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title

• G06F18/2411

  based on the proximity to a decision surface, e.g. support vector machines · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06K9/00255

  Physics · mapped topic

• G06K9/6269

  Physics · mapped topic