Virtual avatar animation based on facial feature movement

Patent metadata

Field	Value
Publication number	US-11107261-B2
Application number	US-201916663062-A
Country	US
Kind code	B2
Filing date	Oct 24, 2019
Priority date	Jan 18, 2019
Publication date	Aug 31, 2021
Grant date	Aug 31, 2021

Abstract

The present disclosure generally relates to displaying visual effects such as virtual avatars. An electronic device having a camera and a display apparatus displays a virtual avatar that changes appearance in response to changes in a face in a field of view of the camera. In response to detecting changes in one or more physical features of the face in the field of view of the camera, the electronic device modifies one or more features of the virtual avatar.

First claim

What is claimed is:

1. An electronic device, comprising:
a display apparatus;
one or more cameras;
one or more processors; and
memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
displaying, via the display apparatus, a virtual avatar having a plurality of avatar features that change appearance in response to detected changes in pose of a face in a field of view of the one or more cameras, the plurality of avatar features including:
a first avatar feature;
a second avatar feature different from the first avatar feature; and
a third avatar feature different from the first and second avatar features;

while the face is detected in the field of view of the one or more cameras, the face including a plurality of detected facial features including a first facial feature and a second facial feature different from the first facial feature, detecting movement of one or more facial features of the face; and
in response to detecting the movement of the one or more facial features:
in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature in a first direction:
moving the first avatar feature in the first direction based on the detected movement of the first facial feature in the first direction; and
moving the second avatar feature and the third avatar feature in a second direction different from the first direction and based on the detected movement of the first facial feature in the first direction, wherein moving the second avatar feature and the third avatar feature in the second direction based on the detected movement of the first facial feature includes:
 in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature by a first magnitude, moving the second avatar feature and the third avatar feature by a first amount of movement; and
 in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature by a second magnitude different from the first magnitude, moving the second avatar feature and the third avatar feature by a second amount of movement that is different from the first amount of movement; and

in accordance with a determination that the detected movement of the one or more facial features includes movement of the second facial feature:
moving the second avatar feature based on at least one of a direction and a magnitude of movement of the second facial feature.

2. The electronic device of claim 1 , the one or more programs further including instructions for:
in response to detecting the movement of the one or more facial features and in accordance with a determination that the detected movement of the one or more facial features includes movement of the second facial feature without more than a first threshold amount of movement of the first facial feature:
moving the second avatar feature based on at least one of the direction and the magnitude of movement of the second facial feature; and
forgoing moving the first and third avatar features.

3. The electronic device of claim 1 , the one or more programs further including instructions for:
in response to detecting the movement of the one or more facial features and in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature without more than a second threshold amount of movement of the second facial feature and a third facial feature:
moving the second and third avatar features based on the detected movement of the first facial feature.

4. The electronic device of claim 1 , the one or more programs further including instructions for:
in response to detecting the movement of the one or more facial features and in accordance with a determination that the detected movement of the one or more facial features includes movement of both the first and second facial features:
moving the first avatar feature based on the detected movement of the first facial feature; and
moving the second avatar feature based on the detected movement of the first and second facial features.

5. The electronic device of claim 1 , wherein moving the first avatar feature in the first direction based on the detected movement of the first facial feature in the first direction includes:
in accordance with a determination that the detected movement of the first facial feature is less than or equal to a second threshold amount of movement of the first facial feature, moving the first avatar feature in the first direction by a first amount determined based on the detected movement of the first facial feature; and
in accordance with a determination that the detected movement of the first facial feature is greater than the second threshold amount of movement of the first facial feature:
moving the first avatar feature in the first direction by a second amount of movement greater than the first amount of movement, and
displaying a first subfeature of the first avatar feature that was not displayed when the movement of the first facial feature was less than the second threshold amount of movement.

6. The electronic device of claim 1 , wherein:
the first facial feature is a jaw of the face;
the first avatar feature is an avatar jaw feature;
the second avatar feature is one or more avatar eyes; and
the third avatar feature is an avatar nose.

7. The electronic device of claim 1 , the one or more programs further including instructions for:
in response to detecting the movement of the one or more facial features and in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature in a third direction:
moving the second avatar feature and the third avatar feature in a fourth direction different from the third direction and based on the detected movement of the first facial feature in the third direction.

8. The electronic device of claim 7 , the one or more programs further including instructions for:
further in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature in the third direction:
moving the first avatar feature in the third direction based on the detected movement of the first facial feature in the third direction.

9. The electronic device of claim 1 , the one or more programs further including instructions for:
further in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature by the first magnitude, moving the first avatar feature by a third amount of movement; and
further in accordance with a determination that the detected movement of the one or more facial features includes movement of the first facial feature by the second magnitude, moving the first avatar feature by a fourth amount of movement different from the third amount of movement.

10. The electronic device of claim 1 , the one or more programs further including instructions for:
detecting a change in pose of the face within the field of view of the one or more cameras; and
in response to detecting the change in pose of the face, changing an appearance of the virtual avatar, including:
in accordance with a determination that the change in pose of the face includes a first type of change in pose of the face, changing the appearance of the virtual avatar includes moving a first portion of the virtual avatar relative

Assignees

• Apple Inc

Inventors

• Scapel Nicolas
• Barlier Guillaume Pierre André
• Guzman Aurelio
• Rickwald Jason

Classifications

• G06V40/176

  Dynamic expression · CPC title

• G06V40/169

  Holistic features and representations, i.e. based on the facial image taken as a whole · CPC title

• G06V40/166

  using acquisition arrangements · CPC title

• G06V40/167

  using comparisons between temporally consecutive images · CPC title

• A63F13/213

  comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) · CPC title