Electronic device and method for controlling the electronic device thereof based on determining intent of a user speech in a first language machine translated into a predefined second language

Patent metadata

Field	Value
Publication number	US-11501089-B2
Application number	US-202016838447-A
Country	US
Kind code	B2
Filing date	Apr 2, 2020
Priority date	Jun 5, 2019
Publication date	Nov 15, 2022
Grant date	Nov 15, 2022

Abstract

An electronic device and a method for controlling the electronic device thereof are provided. The electronic device includes a memory storing instructions, and a processor configured to control the electronic device by executing the instructions stored in the memory, and the processor is configured to, based on a user's speech being input, acquire a first sentence in a first language corresponding to the user's speech through a speech recognition model corresponding to a language of the user's speech, acquire a second sentence in a second language corresponding to the first sentence in the first language through a machine translation model trained to translate a plurality of languages into the predefined second language, and acquire a control instruction of the electronic device corresponding to the acquired second sentence or acquire a response to the second sentence through a natural language understanding model trained based on the second language.

First claim

What is claimed is:

1. An electronic device comprising:
a memory storing at least one instruction; and a processor, operatively coupled to the memory, and configured to control the electronic device by executing the at least one instruction stored in the memory,
wherein the processor is configured by the at least one instruction to:
based on a user's speech being input, acquire a first sentence in a first language corresponding to the user's speech through a speech recognition model corresponding to a language of the user's speech,
acquire a plurality of candidate sentences corresponding to the first sentence and reliability values corresponding to the plurality of candidate sentences through a machine translation model,
determine whether intent in a first-priority sentence is identified by inputting the first-priority sentence having a highest reliability value among the plurality of candidate sentences to the natural language understanding model trained based on a second language,
and based on determining that the intent in the first-priority sentence is not identified, determine whether intent in a second-priority sentence is identified by inputting the second-priority sentence having a reliability value lower than the reliability value of the first-priority sentence among the plurality of candidate sentences to the natural language understanding model trained based on the second language.

2. The electronic device according to claim 1 ,
wherein the memory comprises a first machine translation model and a first natural language understanding model,
wherein, to acquire the plurality of candidate sentences corresponding to the first sentence and reliability values corresponding to the plurality of candidate sentences, the processor is further configured by the at least one instruction to input the first sentence to the first machine translation model,
wherein the processor is further configured by the at least one instruction to acquire a reliability value of the second sentence in response to the input of the first sentence to the first machine translation model, and
wherein the processor is further configured by the at least one instruction to, based on the reliability value being a threshold value or higher, input the second sentence to the first natural language understanding model.

3. The electronic device according to claim 2 , further comprising:
a communication interface configured to communicate with an external server,
wherein the external server comprises a second machine translation model trained to translate a plurality of languages into the second language, and
wherein the processor is further configured by the at least one instruction to:
based on the reliability value being lower than the threshold value, control the communication interface to transmit the first sentence to the external server, and
based on a third sentence in the second language acquired through the second machine translation model being received from the external server, acquire a control instruction corresponding to the third sentence or acquire a response to the third sentence by inputting the third sentence to the first natural language understanding model.

4. The electronic device according to claim 3 , wherein the first machine translation model is retrained based on the first sentence and the third sentence.

5. The electronic device according to claim 3 , further comprising:
wherein the external server comprises a second natural language understanding model trained based on the second language, and
wherein the processor is further configured by the at least one instruction to, based on the reliability value being lower than the threshold value:
control the communication interface to transmit the first sentence to the external server, and
receive a control instruction corresponding to the third sentence or a response to the third sentence acquired through the second machine translation model and the second natural language understanding model from the external server via the communication interface.

6. The electronic device according to claim 1 , further comprising:
a first database storing an input text and a correction text of the machine translation model which are mapped to each other,
wherein the processor is further configured by the at least one instruction to, based on determining that the intent in the second-priority sentence is identified, map at least one text included in the first sentence and at least one text included in the second-priority sentence as the input text and the correction text in the first database.

7. The electronic device according to claim 6 , further comprising:
a second database storing information regarding a proper noun stored in the electronic device,
wherein the processor is further configured by the at least one instruction to, based on the second sentence including the proper noun stored in the second database:
replace at least one text included in the second sentence with the proper noun in the first language stored in the second database, and
acquire a control instruction of the electronic device corresponding to the replaced at least one text included in the second sentence or acquiring a response to the second sentence.

8. The electronic device according to claim 7 , wherein the processor is further configured by the at least one instruction to:
map the at least one text included in the second sentence and the proper noun as an input text and a correction text, and
store the mapping of the at least one text included in the second sentence and the proper noun in the first database.

9. The electronic device according to claim 1 , wherein the machine translation model is trained by performing multi-task learning (MTL) by using the second language as a common parameter.

10. A method for controlling an electronic device, the method comprising:
based on a user's speech being input, acquiring a first sentence in a first language corresponding to the user's speech through a speech recognition model corresponding to a language of the user's speech;
acquiring a plurality of candidate sentences corresponding to the first sentence and reliability values corresponding to the plurality of candidate sentences through a machine translation model;
determining whether intent in a first-priority sentence is identified by inputting the first-priority sentence having a highest reliability value among the plurality of candidate sentences to the natural language understanding model trained based on a second language: and
based on determining that the intent in the first-priority sentence is not identified, determining whether intent in a second-priority sentence is identified by inputting the second priority sentence having a reliability value lower than the reliability value of the first priority sentence among the plurality of candidate sentences to the natural language understanding model trained based on the second language.

11. The method according to claim 10 ,
wherein a memory of the electronic device comprises a first machine translation model and a first natural language understanding model,
wherein the plurality of candidate sentences corresponding to the first sentence and reliability values corresponding to the plurality of candidate sentences comprises:
inputting the first sentence to the first machine translation model,

wherein the method further comprises acquiring a reliability value of the second sentence in response to the inputting of the first sentence to the first machine translation model, and
wherein the method further comprises:
based on the reliability value being a threshold value or higher, inputting the second

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Jiwan
• Kang Seungsoo
• RYU Jongyoub
• Park Soyoon
• Kim Sangha
• Kim Hakjung
• Eom Myungjin

Classifications

• G10L15/18

  using natural language modelling · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06F40/47

  Machine-assisted translation, e.g. using translation memory · CPC title

• G06F40/40 Primary

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title

• G10L15/005

  Language recognition · CPC title