Motion tracking and image recognition of hand gestures to animate a digital puppet, synchronized with recorded audio

Patent metadata

Field	Value
Publication number	US-11049309-B2
Application number	US-201314099691-A
Country	US
Kind code	B2
Filing date	Dec 6, 2013
Priority date	Dec 6, 2013
Publication date	Jun 29, 2021
Grant date	Jun 29, 2021

Abstract

There is provided a system and method for creating a digital puppet show by animating a digital puppet using gestures and sound. The method includes presenting the digital puppet to a user on a display. The method further includes receiving motion data corresponding to a gesture, using a camera, from the user, translating the motion data into digital data using a motion tracking algorithm, and animating the digital puppet, on the display, using the digital data. The method can further include receiving audio data from the user using a microphone and playing the audio data, using a speaker, while animating the digital puppet on the display to create the digital puppet show.

First claim

What is claimed is:

1. A system for animating a three-dimensional (3-D) digital puppet, the system comprising:
a display;
a microphone;
a speaker;
a camera; and
a processor configured to:
present the 3-D digital puppet to a user on the display;
receive, using the camera, motion data corresponding to a gesture made by the user;
receive, using the microphone, audio data corresponding to voice of the user for use in a 3-D digital puppet show;
translate the motion data into digital data using a motion tracking algorithm, the digital data including at least (X,Y,Z) movement data;
create the 3-D digital puppet show using the 3-D digital puppet by:
animating the 3-D digital puppet, on the display, using the digital data;
recording the audio data in real time and along with the animating of the 3-D digital puppet;
searching the audio data for a command;
playing, using the speaker, the audio data at the same time as the animating of the 3-D digital puppet on the display; and
animating facial expressions of the 3-D digital puppet using the command at the same time as the animating the 3-D digital puppet on the display using the digital data translated from the motion data;
wherein the audio data includes a plurality of time marks, and wherein playing the audio data includes synchronizing, using the plurality of time marks, the audio data with the animating of the 3-D digital puppet.

2. The system of claim 1 , wherein before presenting the 3-D digital puppet to the user, the processor is further configured to:
present a plurality of 3-D digital puppets to the user on the display; and
receive a user input for selecting the 3-D digital puppet from the plurality of 3-D digital puppets.

3. The system of claim 1 , wherein the gesture from the user is a hand gesture.

4. The system of claim 3 , wherein a physical accessory corresponding to the 3-D digital puppet is used by the user for the hand gesture.

5. The system of claim 3 , wherein a movement of fingers and a thumb of the hand gesture animates a mouth of the 3-D digital puppet.

6. The system of claim 1 , wherein the digital data further includes rotational data.

7. The system of claim 1 further comprising a memory, wherein the motion tracking algorithm is stored in the memory.

8. The system of claim 1 further comprising a memory, wherein the processor is further configured to record the animation of the 3-D digital puppet in the memory to create the 3-D digital puppet show.

9. The system of claim 1 , wherein the digital data further includes mouth control data.

10. A method for animating a 3-D digital puppet on a device including a processor, a display, a microphone, a speaker and a camera, the method comprising:
presenting the 3-D digital puppet to a user on the display;
receiving, using the camera, motion data corresponding to a gesture made by the user;
receiving, using the microphone, audio data corresponding to voice of the user for use in a 3-D digital puppet show;
translating the motion data into digital data using a motion tracking algorithm, the digital data including at least (X,Y,Z) movement data;
creating a 3-D digital puppet show using the 3-D digital puppet by:
animating the 3-D digital puppet, on the display, using the digital data;
recording the audio data in real time and along with the animating of the 3-D digital puppet;
searching the audio data for a command;
playing, using the speaker, the audio data at the same time as the animating of the 3-D digital puppet on the display; and
animating facial expressions of the 3-D digital puppet using the command at the same time as the animating the 3-D digital puppet on the display using the digital data translated from the motion data;
wherein the audio data includes a plurality of time marks, and wherein playing the audio data includes synchronizing, using the plurality of time marks, the audio data with the animating of the 3-D digital puppet.

11. The method of claim 10 , wherein before presenting the 3-D digital puppet to the user, the method further comprises:
presenting a plurality of 3-D digital puppets to the user on the display; and
receiving a user input for selecting the 3-D digital puppet from the plurality of 3-D digital puppets.

12. The method of claim 10 , wherein the gesture from the user is a hand gesture.

13. The method of claim 12 , wherein a physical accessory corresponding to the 3-D digital puppet is used by the user for the hand gesture.

14. The method of claim 12 , wherein a movement of fingers and a thumb of the hand gesture animates a mouth of the 3-D digital puppet.

15. The method of claim 10 , wherein the digital data further includes rotational data.

16. The method of claim 10 , wherein the motion tracking algorithm is stored in a memory.

17. The method of claim 10 , further comprising:
recording the animation of the 3-D digital puppet in a memory to create the 3-D digital puppet show.

18. The method of claim 10 , wherein the digital data further includes mouth control data.

Assignees

• Disney Entpr Inc

Inventors

• Murdock Malcolm
• Hy Michael
• Mkrtchyan Armen

Classifications

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G10L2021/105

  Synthesis of the lips movements from speech, e.g. for talking heads · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title