Dynamic facial feature substitution for video conferencing

Patent metadata

Field	Value
Publication number	US-9277180-B2
Application number	US-201414320233-A
Country	US
Kind code	B2
Filing date	Jun 30, 2014
Priority date	Jun 30, 2014
Publication date	Mar 1, 2016
Grant date	Mar 1, 2016

Abstract

In an approach to determine facial feature substitution in a video conference, a computer receives one or more pre-recorded videos of an attendee of a video conference. The one or more pre-recorded videos include at least one pre-recorded video of an attendee speaking. The computer determines an avatar for use in place of a video of the attendee in the video conference. Furthermore, the computer determines one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the video of the attendee in the video conference. The computer substitutes the one or more portions of the one or more pre-recorded videos into the avatar, the substitution corresponding to the at least one targeted facial feature of the attendee in the video conference.

First claim

What is claimed is:

1. A computer program product for facial feature substitution in a video conference, the computer program product comprising:
one or more computer readable storage devices and program instructions stored on the one or more computer readable storage devices, the program instructions executable by a processor, the program instructions comprising:
program instructions to receive one or more pre-recorded videos of an attendee participating in a video conference, the one or more pre-recorded videos including at least one pre-recorded video of an attendee speaking;
program instructions to determine an avatar for use in place of a live video feed of the attendee in the video conference;
program instructions to determine one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the live video feed of the attendee participating in the video conference; and
program instructions to substitute the one or more portions of the one or more pre-recorded videos into the avatar, the substitution corresponding to the at least one targeted facial feature of the attendee such that the avatar mimics the at least one targeted facial feature of the attendee in the video conference.

2. The computer program product of claim 1 , wherein the program instructions to substitute the one or more portions of the one or more pre-recorded videos into the avatar further comprise:
program instructions to create a co-ordinate map of one or more key facial elements of the attendee in the one or more pre-recorded videos;
program instructions to create a co-ordinate map of one or more key facial elements of the attendee in the video of the attendee in the video conference;
program instructions to match the co-ordinate map of the one or more pre-recorded videos to the co-ordinate map of the video of the attendee in the video conference; and
program instructions to substitute, based, at least in part, on the matched co-ordinate maps, the one or more portions of the one or more pre-recorded videos into the avatar.

3. The computer program product of claim 1 , wherein the program instructions to determine one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the live video feed of the attendee participating in the video conference further comprise program instructions to use facial recognition algorithms to correlate facial expressions and facial movements in the live video feed to the one or more pre-recorded videos.

4. The computer program product of claim 1 , wherein the program instructions to determine one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the live video feed of the attendee participating in the video conference further comprise program instructions to use at least one of natural language processing and speech recognition to match corresponding one or more words in the live video feed of the attendee participating in the video conference to one or more words in the one or more pre-recorded videos.

5. The computer program product of claim 1 , wherein the program instructions to determine one or more portions of the one or more pre-recorded videos of the attendee for at least one targeted facial feature of the attendee in the live video feed of the attendee participating in the video conference further comprise program instructions to use sentiment analysis to correlate a sentiment of the live video feed of the attendee participating in the video conference to a sentiment the one or more portions of the one or more pre-recorded videos.

6. The computer program product of claim 1 , further comprises:
program instructions to monitor the live video feed of the attendee participating in the video conference;
program instructions to determine whether the at least one targeted facial feature of the attendee changed;
responsive to determining the at least one targeted facial feature of the attendee changed, program instructions to determine at least one additional portion of the one or more pre-recorded videos corresponding to the changed at least one targeted facial feature; and
program instructions to substitute the at least one additional portion of the one or more pre-recorded videos into the avatar.

7. A computer system for facial feature substitution in a video conference, the computer system comprising:
one or more computer processors;
one or more computer readable storage devices;
program instructions stored on the computer readable storage devices for execution by at least one of the one or more processors, the program instructions comprising:
program instructions to receive one or more pre-recorded videos of an attendee participating in a video conference, the one or more pre-recorded videos including at least one pre-recorded video of an attendee speaking;
program instructions to determine an avatar for use in place of a live video feed of the attendee in the video conference;
program instructions to determine one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the live video feed of the attendee participating in the video conference; and
program instructions to substitute the one or more portions of the one or more pre-recorded videos into the avatar, the substitution corresponding to the at least one targeted facial feature of the attendee such that the avatar mimics the at least one targeted facial feature of the attendee in the video conference.

8. The computer system of claim 7 , wherein the program instructions to substitute the one or more portions of the one or more pre-recorded videos into the avatar further comprise:
program instructions to create a co-ordinate map of one or more key facial elements of the attendee in the one or more pre-recorded videos;
program instructions to create a co-ordinate map of one or more key facial elements of the attendee in the video of the attendee in the video conference;
program instructions to match the co-ordinate map of the one or more pre-recorded videos to the co-ordinate map of the video of the attendee in the video conference; and
program instructions to substitute, based, at least in part, on the matched co-ordinate maps, the one or more portions of the one or more pre-recorded videos into the avatar.

9. The computer system of claim 7 , wherein the program instructions to determine one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the live video feed of the attendee participating in the video conference further comprise program instructions to use facial recognition algorithms to correlate facial expressions and facial movements in the live video feed to the one or more pre-recorded videos.

10. The computer system of claim 7 , wherein the program instructions to determine one or more portions of the one or more pre-recorded videos of the attendee corresponding to at least one targeted facial feature of the attendee in the live video feed of the attendee participating in the video conference further comprise program instructions to use at least one of natural language processing and speech recognition to match corresponding one or more words in the live video feed of the attendee participating in the video conference to one or more words in the one or more pre-recorded videos.

11. The computer system of claim 7 , wherein the program instructions to determine one

Assignees

• IBM

Inventors

• Cunico Hernan A
• Silva Asima

Classifications

• G06V10/75

  Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• H04N7/152

  Multipoint control units therefor · CPC title

• G06T7/0042

  Physics · mapped topic

• G06T2207/10016

  Video; Image sequence · CPC title