Multilingual semantic search utilizing meta-distillation learning

Patent metadata

Field	Value
Publication number	US-12547901-B2
Application number	US-202318449291-A
Country	US
Kind code	B2
Filing date	Aug 14, 2023
Priority date	Aug 14, 2023
Publication date	Feb 10, 2026
Grant date	Feb 10, 2026

Abstract

The present disclosure relates to systems, non-transitory computer-readable media, and methods for providing multilingual semantic search results utilizing meta-learning and knowledge distillation. For example, in some implementations, the disclosed systems perform a first inner learning loop for a monolingual to bilingual meta-learning task for a teacher model. Additionally, in some implementations, the disclosed systems perform a second inner learning loop for a bilingual to multilingual meta-learning task for a student model. In some embodiments, the disclosed systems perform knowledge distillation based on the first inner learning loop for the monolingual to bilingual meta-learning task and the second inner learning loop for the bilingual to multilingual meta-learning task. Moreover, in some embodiments, the disclosed systems perform an outer learning loop and update parameters of a deep learning language model based on the first inner learning loop, the second inner learning loop, and the knowledge distillation.

First claim

What is claimed is:

1 . A computer-implemented method comprising:
learning parameters of a deep learning language model utilizing meta-learning and knowledge distillation to cause the deep learning language model to perform multilingual search retrieval comprising:
performing a first inner learning loop for a monolingual to bilingual meta-learning task for a teacher model;
performing a second inner learning loop for a bilingual to multilingual meta-learning task for a student model;
performing knowledge distillation based on the first inner learning loop for the monolingual to bilingual meta-learning task and the second inner learning loop for the bilingual to multilingual meta-learning task; and
performing an outer learning loop and updating parameters of the deep learning language model based on the first inner learning loop, the second inner learning loop, and the knowledge distillation.

2 . The computer-implemented method of claim 1 , wherein performing the first inner learning loop comprises:
updating parameters of the teacher model based on a support loss for the monolingual to bilingual meta-learning task; and generating a first query loss based on the updated parameters of the teacher model.

3 . The computer-implemented method of claim 2 , wherein performing the second inner learning loop comprises:
updating parameters of the student model based on a support loss for the bilingual to multilingual meta-learning task; and generating a second query loss based on the updated parameters of the student model.

4 . The computer-implemented method of claim 3 , wherein performing the knowledge distillation comprises comparing the first query loss and the support loss for the bilingual to multilingual meta-learning task.

5 . The computer-implemented method of claim 1 , wherein performing the outer learning loop comprises generating a task loss by combining a plurality of first query losses from the first inner learning loop and a plurality of second query losses from the second inner learning loop.

6 . The computer-implemented method of claim 5 , wherein updating the parameters of the deep learning language model comprises evaluating a gradient of the task loss and a knowledge distillation loss.

7 . A system comprising:
one or more memory devices comprising a multilingual deep learning language model having parameters learned utilizing meta-learning and knowledge distillation, wherein the parameters of the multilingual deep learning language model are learned by generating a knowledge distillation loss based on a query loss for a monolingual to bilingual meta-learning task and a support loss for a bilingual to multilingual meta-learning task; and one or more processors configured to cause the system to:
receive, from a client device, a user interaction requesting a semantic search result;
in response to the user interaction, generate a multilingual sentence-level search result utilizing the multilingual deep learning language model having parameters learned utilizing meta-learning and knowledge distillation; and
provide the multilingual sentence-level search result for display via the client device.

8 . The system of claim 7 , wherein the parameters of the multilingual deep learning language model are learned by performing a first inner learning loop for a monolingual to bilingual meta-learning task to generate a query loss for the monolingual to bilingual meta-learning task.

9 . The system of claim 7 , wherein the parameters of the multilingual deep learning language model are learned by performing a second inner learning loop for a bilingual to multilingual meta-learning task to generate a query loss for the bilingual to multilingual meta-learning task.

10 . The system of claim 7 , wherein generating the knowledge distillation loss comprises comparing the query loss for the monolingual to bilingual meta-learning task and the support loss for the bilingual to multilingual meta-learning task.

11 . The system of claim 7 , wherein the one or more processors are configured to cause the system to receive the user interaction requesting a semantic search result by receiving an interaction in a first language, and
wherein the one or more processors are configured to cause the system to provide the multilingual sentence-level search result by providing a search result in a plurality of languages comprising a second language and a third language.

12 . The system of claim 7 , wherein utilizing the multilingual deep learning language model comprises utilizing a transformer-based neural network.

13 . The system of claim 7 , wherein the one or more processors are configured to cause the system to generate the multilingual sentence-level search result by:
encoding a question utilizing the multilingual deep learning language model to generate an encoded question; encoding a candidate answer and context utilizing the multilingual deep learning language model to generate an encoded answer; and comparing the encoded question and the encoded answer.

14 . A non-transitory computer-readable medium storing executable instructions that, when executed by a processing device, cause the processing device to perform operations comprising:
generating, utilizing a first inner loop of an alignment learner, a first query loss for a monolingual to bilingual meta-learning task for a teacher model; generating, utilizing a second inner loop of the alignment learner, a second query loss for a bilingual to multilingual meta-learning task for a student model; generating a knowledge distillation loss for the teacher model and the student model; and updating parameters of a deep learning language model, utilizing an outer loop of the alignment learner, based on the first query loss, the second query loss, and the knowledge distillation loss.

15 . The non-transitory computer-readable medium of claim 14 , wherein generating the first query loss comprises utilizing the teacher model to evaluate a query set of the monolingual to bilingual meta-learning task.

16 . The non-transitory computer-readable medium of claim 14 , wherein generating the second query loss comprises utilizing the student model to evaluate a query set of the bilingual to multilingual meta-learning task.

17 . The non-transitory computer-readable medium of claim 14 , wherein generating the knowledge distillation loss comprises comparing the first query loss for the monolingual to bilingual meta-learning task and a support loss for the bilingual to multilingual meta-learning task.

18 . The non-transitory computer-readable medium of claim 14 , wherein the operations further comprise determining a task loss by combining the first query loss and the second query loss.

19 . The non-transitory computer-readable medium of claim 18 , wherein updating the parameters of the deep learning language model comprises determining a gradient of the task loss combined with the knowledge distillation loss.

20 . The non-transitory computer-readable medium of claim 14 , wherein the operations further comprise:
sampling a first batch of monolingual to bilingual meta-learning tasks for the first inner loop of the alignment learner; sampling a second batch of bilingual to multilingual meta-learning tasks for the second inner loop of the alignment learner; and updating the parameters of the deep learning language model by utilizing the outer loop of the alignment learner to update parameters of the student model.

Assignees

• Adobe Inc

Inventors

• M'Hamdi Meryem
• Yoon Seunghyun
• Dernoncourt Franck
• Bui Trung

Classifications

• G06F40/30

  Semantic analysis · CPC title

• G06F40/40 Primary

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/096 Primary

  Transfer learning · CPC title