Method and apparatus for sound object following

Patent metadata

Field	Value
Publication number	US-11277702-B2
Application number	US-202016812183-A
Country	US
Kind code	B2
Filing date	Mar 6, 2020
Priority date	Mar 8, 2019
Publication date	Mar 15, 2022
Grant date	Mar 15, 2022

Abstract

The present disclosure relates to a method and apparatus for processing a multimedia signal. More specifically, the present disclosure relates to a method comprising obtaining at least one video object from the multimedia signal and at least one audio object from the multimedia signal, extracting video feature information for the at least one video object and audio feature information for the at least one audio object, and determining a correlation between the at least one video object and the at least one audio object through an object matching engine based on the video feature information and the audio feature information, and an apparatus therefor.

First claim

What is claimed is:

1. A method of processing a multimedia signal by an apparatus, the method comprising:
obtaining at least one video object from the multimedia signal and at least one audio object from the multimedia signal, wherein the at least one video object represents a face;
extracting video feature information for the at least one video object and audio feature information for the at least one audio object;
determining a correlation between the at least one video object and the at least one audio object through an object matching engine based on the video feature information and the audio feature information; and
based on the correlation, controlling output of left and right audio signals for sound image localization at a screen location of the at least one video object with respect to a listener,
wherein the video feature information is extracted based on:
a ratio between a distance between a top boundary of a first rectangular region including the face and a top boundary of a second rectangular region including lips within the first rectangular region, a height of the second rectangular region, and a distance between a bottom boundary of the second rectangular region and a bottom boundary of the first rectangular region; and
a ratio between a distance between a left boundary of the second rectangular region and a left boundary of the first rectangular region, a width of the second rectangular region, and a distance between a right boundary of the second rectangular region and a right boundary of the first rectangular region.

2. The method of claim 1 , wherein determining the correlation between the at least one video object and the at least one audio object comprises:
obtaining information about a relationship between each of the at least one video object and a specific audio object of the at least one audio object through the object matching engine based on the video feature information and the audio feature information; and
determining, from among the at least one video object, a specific video object related to the specific audio object based on the information about the relationship between each of the at least one video object and the specific audio object.

3. The method of claim 2 , wherein determining the specific video object related to the specific audio object comprises:
based on a value of the information about the relationship being greater than a predetermined value, determining a video object related to the value of the information about the relationship as the specific video object.

4. The method of claim 3 , wherein determining the specific video object related to the specific audio object further comprises:
based on the least one video object comprising a plurality of video objects, determining, from among the plurality of video objects, a video object related to a greatest value of the information about the relationship as the specific video object.

5. The method of claim 2 , wherein determining the specific video object related to the specific audio object comprises:
based on a value of the information about the relationship being less than a predetermined value, determining a video object related to the value of the information about the relationship as the specific video object.

6. The method of claim 5 , wherein determining the specific video object related to the specific audio object further comprises:
based on the least one video object comprising a plurality of video objects, determining, from the plurality of video objects, a video object related to a smallest value of the information about the relationship as the specific video object.

7. The method of claim 2 , wherein the information about the relationship has a real number value.

8. The method of claim 1 , wherein the video feature information is extracted based on a vertical length and a horizontal length of a lip skeleton.

9. The method of claim 1 , wherein the audio feature information is extracted based on linear prediction coding (LPC).

10. The method of claim 1 , wherein the audio feature information is extracted based on a log-Mel filters-of-bank.

11. The method of claim 1 , wherein the audio feature information is extracted based on Mel-frequency cepstral coefficients (MFCC).

12. The method of claim 1 , wherein the audio feature information comprises onset information about the at least one audio object.

13. The method of claim 1 , wherein the object matching engine comprises a model trained based on learning.

14. An apparatus configured to process a multimedia signal, the apparatus comprising:
a memory storing instructions; and
at least one processor operatively coupled to the memory and configured to, when executing the instructions, implement operations comprising:
obtaining at least one video object from the multimedia signal and at least one audio object from the multimedia signal, wherein the at least one video object represents a face;
extracting video feature information for the at least one video object and audio feature information for the at least one audio object;
determining a correlation between the at least one video object and the at least one audio object through an object matching engine based on the video feature information and the audio feature information; and
based on the correlation, controlling output of left and right audio signals for sound image localization at a screen location of the at least one video object with respect to a listener,
wherein the video feature information is extracted based on:
a ratio between a distance between a top boundary of a first rectangular region including the face and a top boundary of a second rectangular region including lips within the first rectangular region, a height of the second rectangular region, and a distance between a bottom boundary of the second rectangular region and a bottom boundary of the first rectangular region; and
a ratio between a distance between a left boundary of the second rectangular region and a left boundary of the first rectangular region, a width of the second rectangular region, and a distance between a right boundary of the second rectangular region and a right boundary of the first rectangular region.

15. The apparatus of claim 14 , wherein determining the correlation between the at least one video object and the at least one audio object comprises:
obtaining information about a relationship between each of the at least one video object and a specific audio object of the at least one audio object through the object matching engine based on the video feature information and the audio feature information; and
determining, from among the at least one video object, a specific video object related to the specific audio object based on the information about the relationship between each of the at least one video object and the specific audio object.

16. The apparatus of claim 15 , wherein determining the specific video object related to the specific audio object comprises:
based on a value of the information about the relationship being greater than a predetermined value, determining a video object related to the value of the information about the relationship as the specific video object.

17. The apparatus of claim 16 , wherein determining the specific video object related to the specific audio object further comprises:
based on the least one video object comprising a plurality of video objects, determining, from among the plurality of video objects, a video object related to a greatest value of the information about the relationship as the specific video obj

Assignees

• Lg Electronics Inc

Inventors

• Jung Sungwon
• Choi Tacksung
• Kang Donghyun
• Lee Seungsu
• Cho Taegil

Classifications

• H04S7/30 Primary

  Control circuits for electronic adaptation of the sound field · CPC title

• G06V10/806

  of extracted features · CPC title

• G06V40/171 Primary

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06V10/75

  Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries · CPC title