Dynamic facial feature substitution for video conferencing

Patent metadata

Field	Value
Publication number	US-9332227-B2
Application number	US-201514623079-A
Country	US
Kind code	B2
Filing date	Feb 16, 2015
Priority date	Jun 30, 2014
Publication date	May 3, 2016
Grant date	May 3, 2016

Abstract

In an approach to determine facial feature substitution in a video conference, a computer receives one or more pre-recorded videos of an attendee of a video conference. The one or more pre-recorded videos include at least one pre-recorded video of an attendee speaking. The computer determines an avatar for use in place of a video of the attendee in the video conference. Furthermore, the computer determines one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the video of the attendee in the video conference. The computer substitutes the one or more portions of the one or more pre-recorded videos into the avatar, the substitution corresponding to the at least one targeted facial feature of the attendee in the video conference.

First claim

What is claimed is:

1. A method for facial feature substitution in a video conference, the method comprising:
receiving, by one or more computing devices, one or more pre-recorded videos of an attendee of a video conference, the one or more pre-recorded videos including at least one pre-recorded video of an attendee speaking;
determining, by one or more computing devices, an avatar for use in place of a video of the attendee in the video conference;
determining, by one or more computing devices, one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the video of the attendee in the video conference; and
substituting, by one or more computing devices, the one or more portions of the one or more pre-recorded videos into the avatar, the substitution corresponding to the at least one targeted facial feature of the attendee.

2. The method of claim 1 , wherein substituting, by one or more computing devices, the one or more portions of the one or more pre-recorded videos into the avatar further comprise:
creating, by one or more computing devices, a co-ordinate map of one or more key facial elements of the attendee in the one or more pre-recorded videos;
creating, by one or more computing devices, a co-ordinate map of one or more key facial elements of the attendee in the video of the attendee in the video conference;
matching, by one or more computing devices, the co-ordinate map of the one or more pre-recorded videos to the co-ordinate map of the video of the attendee in the video conference; and
substituting, by one or more computing devices, based, at least in part, on the matched co-ordinate maps, the one or more portions of the one or more pre-recorded videos into the avatar.

3. The method of claim 1 , wherein determining, by one or more computing devices, the one or more portions of the one or more pre-recorded videos corresponding to the at least one targeted facial feature of the attendee in the video of the attendee in the video conference further comprises using one or more facial recognition algorithms to correlate facial expressions and facial movements in the video to the one or more pre-recorded videos.

4. The method of claim 1 , wherein determining, by one or more computing devices, the one or more portions of the one or more pre-recorded videos corresponding to the at least one targeted facial feature of the attendee in the video of the attendee in the video conference further comprises using at least one of natural language processing and speech recognition to match corresponding one or more words in the video of the attendee to one or more words in the one or more pre-recorded videos.

5. The method of claim 1 , wherein determining, by one or more computing devices, the one or more portions of the one or more pre-recorded videos corresponding to the at least one targeted facial feature of the attendee in the video of the attendee in the video conference further comprises using sentiment analysis to correlate the sentiment of the video of the attendee to sentiment of the one or more portions of the one or more pre-recorded videos.

6. The method of claim 1 , wherein determining, by one or more computing devices, the one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the video of the attendee in the video conference further comprises using three dimensional facial recognition algorithms to correlate the video of the attendee with one or more portions of the one or more pre-recorded videos.

7. The method of claim 1 , wherein the one or more pre-recorded videos of the attendee further include the attendee exhibiting one or more facial expressions.

8. The method of claim 1 , further comprising:
monitoring, by one or more computing devices, the video of the attendee in a video conference;
determining, by one or more computing devices, whether the at least one targeted facial feature of the attendee changed;
responsive to determining, by one or more computing devices, the at least one targeted facial feature of the attendee changed, determining at least one additional portion of the one or more pre-recorded videos of the attendee corresponding to the changed at least one targeted facial feature; and
substituting, by one or more computing devices, the at least one additional portion of the one or more pre-recorded videos into the avatar.

Assignees

• IBM

Inventors

• Cunico Hernan A
• Silva Asima

Classifications

• G06V10/75

  Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06T7/0042

  Physics · mapped topic

• H04L65/601

  Electricity · mapped topic

• G06T2207/10016

  Video; Image sequence · CPC title