Voice recognition method and apparatus using video recognition

Patent metadata

Field	Value
Publication number	US-10049665-B2
Application number	US-201313950933-A
Country	US
Kind code	B2
Filing date	Jul 25, 2013
Priority date	Jul 26, 2012
Publication date	Aug 14, 2018
Grant date	Aug 14, 2018

Abstract

Provided are a method and an apparatus for performing exact start and end recognition of voice based on video recognition. The method includes determining whether a speech starts based on at least one of first video and audio data before conversion into a voice recognition mode, converting into the voice recognition mode and generating second audio data including a voice command, when it is determined that speech starts, and determining whether the speech is terminated based on at least one of second video and audio data after conversion into the voice recognition mode.

First claim

What is claimed is:

1. A method of recognizing a voice command by an electronic device, the method comprising:
analyzing first video data and first audio data to determine a start of speech, the first video data being a mouth shape;
when it is determined that results of both analyzing the first video data and analyzing the first audio data indicate the speech starts, converting a mode to a voice recognition mode;
when it is determined that the result of analyzing the first audio data indicates that the speech starts,
attenuating a threshold of video data for determining whether the speech starts and re-determining whether the speech starts;
when it is determined that the result of analyzing the first video data indicates that the speech starts, attenuating a threshold of audio data for determining whether the speech starts and re-determining whether the speech starts;
simultaneously displaying at least one user-selectable icon operatively connected to an executable function of the electronic device and a voice command corresponding to each of the at least one user-selectable icon in the voice recognition mode; and
determining whether the speech is terminated based on second video data and second audio data.

2. The method of claim 1 , further comprising removing display of the voice command after determining whether the speech is terminated.

3. The method of claim 2 , wherein determining whether the speech starts comprises identifying a lip of the user from the first video data and determining whether the speech starts based on whether the lip is in an open position.

4. The method of claim 1 , wherein determining whether the speech is terminated comprises determining that the speech is terminated when a voice of the user is not included in the second audio data for at least a preset time.

5. The method of claim 1 , wherein determining whether the speech is terminated comprises determining that the speech is terminated when a face of the user is not included in the second video data.

6. An electronic device which recognizes a voice command, the electronic device comprising:
an audio processor which collects and records a voice input;
a camera unit which collects and records a video input;
a display unit which displays a screen; and
a controller which:
analyzes first video data and first audio data to determine the start of speech, the first video data being a mouth shape;
when it is determined that results of both analyzing the first video data and analyzing the first audio data indicate that the speech starts, converts a mode to a voice recognition mode;
when it is determined that the result of analyzing the first audio data indicates that the speech starts, attenuates a threshold of video analysis for determining whether the speech starts and re-determines whether the speech starts;
when it is determined that the result of analyzing the first video data indicates that the speech starts, attenuates a threshold of audio analysis for determining whether the speech starts and re-determines whether the speech starts;
when it is determined that the speech starts, controls the display unit to simultaneously display at least one user-selectable icon operatively connected to an executable function of the electronic device and a voice command corresponding to each of the at least one user-selectable icon; and
determines whether the speech is terminated based on second video data and second audio data.

7. The electronic device of claim 6 ,
wherein the controller controls the display unit to remove display of the voice command when the speech is terminated.

8. The electronic device of claim 7 , wherein the controller identifies a lip of a user from the first video data, and determines whether the speech starts based on whether the lip is in an open position.

9. The electronic device of claim 6 , wherein the controller determines that the speech is terminated when a voice of the user is not included in the second audio data for at least a preset time.

10. The electronic device of claim 6 , wherein the controller determines that the speech is terminated when a face of the user is not included in the second video data.

11. A non-transitory processor-readable medium for storing a platform for recognizing a voice command, the platform comprising:
a user interface framework which displays a screen, wherein the user interface framework simultaneously displays at least one user-selectable icon operatively connected to an executable function of the electronic device and a voice command corresponding to each of the at least one user-selectable icon;
a multi-media framework which collects and records voice input and video input to generate first video data and first audio data before conversion into a voice recognition mode, and generates second video data and second audio data after conversion into the voice recognition mode; and
a voice framework which:
analyzes the first video data and the first audio data to determine the start of speech, the first video data being a mouth shape;
when it is determined that results of both analyzing the first video data and analyzing the first audio data indicate that speech starts, converts a mode to a voice recognition mode;
when it is determined that the result of analyzing the first audio data indicates that the speech starts, attenuates a threshold of video analysis for determining whether the speech starts and re-determines whether the speech starts;
when it is determined that the result of analyzing the first video data indicates that the speech starts, attenuates a threshold of audio analysis for determining whether the speech starts and re-determines whether the speech starts;
when it is determined that the speech starts, requests voice command configuration to the user interface framework, and
determines whether the speech is terminated based on the second video data and the second audio data.

12. The medium of claim 11 ,
wherein the user interface framework removes display of the voice command when the voice framework determines that the speech is terminated.

13. The medium of claim 11 , wherein the voice framework identifies a lip of a user from the first video data to determine whether the speech starts based on whether the lip is in an open position, and determines that the speech is terminated when a voice of the user is not included in the second audio data for at least a preset time.

14. The medium of claim 13 , wherein the voice framework determines that the speech is terminated when a face of the user is not included in the second video data.

Assignees

• Samsung Electronics Co Ltd
• Samsung Electronics Co Ltd

Inventors

• Lee Dongyeol
• Suh Sangbum

Classifications

• G10L25/87 Primary

  Detection of discrete points within a voice signal · CPC title

• G10L15/22

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G10L15/25 Primary

  using position of the lips, movement of the lips or face analysis · CPC title

• G06V10/10 Primary

  Image acquisition (document image scanning and transmission H04N1/00; control of digital cameras H04N23/60) · CPC title

• G10L15/00

  Speech recognition (G10L17/00 takes precedence) · CPC title