Method and apparatus for generating interaction record, and device and medium

Patent metadata

Field	Value
Publication number	US-12087285-B2
Application number	US-202217881999-A
Country	US
Kind code	B2
Filing date	Aug 5, 2022
Priority date	Apr 30, 2020
Publication date	Sep 10, 2024
Grant date	Sep 10, 2024

Abstract

A method and apparatus for generating an interaction record, and a device and a medium are provided. The method includes: firstly, from a multimedia data stream, collecting behavior data, represented by the multimedia data stream, of a user, wherein the behavior data includes voice information and/or operation information; and then, on the basis of the behavior data, generating interaction record data corresponding to the behavior data. According to the technical solution, by means of collecting voice information and/or operation information from a multimedia data stream, and generating interaction record data on the basis of the voice information and the operation information, an interacting user can determine interaction information by using the interaction record data, and the interaction efficiency of the interacting user is improved, thereby also improving the user experience.

First claim

The invention claimed is:

1. A method for generating an interactive record, wherein at least two users that are located remotely are participants in a multimedia conference, a live video broadcast, or a group chat via respective client devices, the method comprising:
collecting, from a multimedia data stream, behavior data of one of the at least two users represented by the multimedia data stream, wherein the behavior data comprises operation information;
generating interactive record data corresponding to the behavior data, based on the behavior data, wherein the interactive record data is an interactive record text corresponding to the behavior data, and the interactive record data comprises a narrative description of an operation being performed in response to the operation information in addition to displaying the operation information; and
sending the interactive record data to a target client, to display the interactive record data on the target client, wherein both the operation information and the narrative description of the operation are displayed at the target client.

2. The method according to claim 1 , wherein the generating interactive record data corresponding to the behavior data comprises:
determining an operation object and an operation behavior in the operation information; and generating the interactive record data based on an association relationship between the operation object and the operation behavior.

3. The method according to claim 2 , wherein in a case that the operation information comprises document sharing operation information, the operation object comprises a shared document, the operation behavior comprises a behavior to share a document, and the generating interactive record data corresponding to the behavior data based on the behavior data comprises:
determining a document sharing address and/or a storage address associated with the shared document, based on the shared document; and generating the interactive record data based on the shared address and/or the storage address.

4. The method according to claim 2 , wherein in a case that the operation information comprises screen sharing operation information, the operation object comprises a shared screen, the operation behavior comprises a behavior to share the shared screen, and the generating interactive record data corresponding to the behavior data based on the behavior data comprises:
determining identification information in the shared screen based on the shared screen; and generating the interactive record data based on the identification information.

5. The method according to claim 1 , wherein the behavior data further comprises speech information, and the generating interactive record data corresponding to the behavior data based on the behavior data comprises:
performing voiceprint recognition on the speech information, to determine a speaking user corresponding to the speech information;
performing speech recognition on the speech information, to obtain a speech recognition result; and
generating the interactive record data corresponding to the behavior data, based on an association between the speaking user and the speech recognition result.

6. The method according to claim 2 , wherein the collecting behavior data of a user represented by the multimedia data stream comprises:
collecting speech information and operation information in a screen recording video.

7. The method according to claim 6 , wherein the generating interactive record data corresponding to the behavior data based on the behavior data comprises:
generating the interactive record data corresponding to the behavior data by performing information extraction on the operation object in the operation information.

8. The method according to claim 2 , wherein in a case that the multimedia data stream is a data stream generated based on a real-time interactive interface, the collecting, from a multimedia data stream, behavior data of a user represented by the multimedia data stream comprises:
collecting behavior data of each user based on request information for generating an interactive record, in response to a reception of the request information.

9. The method according to claim 8 , wherein the behavior data further comprises speech information, and the collecting behavior data of a user represented by the multimedia data stream comprises at least one of:
receiving speech information of each user collected by a client; and
receiving request information corresponding to a trigger operation, and determining operation information corresponding to the request information.

10. The method according to claim 8 , wherein in a case that the behavior data comprises the operation information, the determining an operation object and an operation behavior in the operation information, and generating the interactive record data based on an association relationship between the operation object and the operation behavior comprises:
acquiring a shared document and associated information corresponding to the shared document, in response to a detection of a trigger operation for sharing a document;
determining the operation information based on the trigger operation, the shared document and the associated information, wherein the associated information comprises a shared link of the shared document and/or a storage address of the shared document; and
generating the interactive record data corresponding to the behavior data based on the operation information.

11. The method according to claim 8 , wherein in a case that the behavior data comprises the operation information, the determining an operation object and an operation behavior in the operation information, and generating the interactive record data based on an association relationship between the operation object and the operation behavior comprises:
identifying identification information in a shared screen, in response to a detection of a trigger operation for sharing a screen;
determining the operation information based on the identification information, the trigger operation and a video frame of the shared screen; and
generating the interactive record data corresponding to the behavior data based on the operation information;
wherein the identification information comprises a link in the shared screen.

12. The method according to claim 1 , wherein the behavior data further comprises speech information, the generating interactive record data corresponding to the behavior data based on the behavior data comprises:
performing speech recognition on the speech information; and generating the interactive record data based on an obtained speech recognition result.

13. The method according to claim 12 , wherein the performing speech recognition on the speech information, and generating the interactive record data based on an obtained speech recognition result comprises:
determining a target language type of a speaking user to which the speech information belongs; and
processing the speech information in the behavior data based on the target language type, to generate the interactive record data.

14. The method according to claim 13 , wherein the determining a target language type of a speaking user to which the speech information belongs comprises:
determining the target language type based on a language type of a speaking user to which a current client belongs.

15. The method according to claim 14 , wherein the language type of the speaking user to which the current client belongs is determined by at least one of:
determining the language type of the user by performing language type recogn

Assignees

• Beijing Bytedance Network Tech Co Ltd

Inventors

• Yang Jingsheng
• Chen Kojung
• Zhao Li
• Han Xiao
• Shi Yin

Classifications

• G10L25/72

  for transmitting results of analysis · CPC title

• G10L25/57

  for processing of video signals · CPC title

• G10L15/005 Primary

  Language recognition · CPC title

• H04L12/1827

  Network arrangements for conference optimisation or adaptation · CPC title

• H04L12/1831

  Tracking arrangements for later retrieval, e.g. recording contents, participants activities or behavior, network status · CPC title