Generating suggested document edits from recorded media using artificial intelligence

Patent metadata

Field	Value
Publication number	US-2020293608-A1
Application number	US-201916355190-A
Country	US
Kind code	A1
Filing date	Mar 15, 2019
Priority date	Mar 15, 2019
Publication date	Sep 17, 2020
Grant date	—

Abstract

Artificial intelligence is introduced into document review to identify content suggestions from input to generate suggested annotations for the reviewed document. An approach is provided for receiving an electronic document that contains original content from an original electronic document for review and electronic mark-ups provided by a first user. One or more electronic mark-ups that represent content suggestions proposed by the first user are identified from the electronic document. For each electronic mark-up of the one or more electronic mark-ups identified a document portion of the original content that corresponds to the electronic mark-up is identified, and an annotation is generated for the electronic mark-up comprising the electronic mark-up and a first user ID for the first user and associating the annotation to the document portion identified. The original content with one or more annotations generated from the one or more electronic mark-ups is displayed, in electronic form, within a display window.

First claim

What is claimed is:

1 . An apparatus comprising:
one or more processors; and one or more memories storing instructions which, when processed by the one or more processors, cause:
receiving one or more media content items representing captured content from a discussion of one or more electronic documents by one or more users;
identifying, from the one or more media content items, portions of media content corresponding to content suggestions for the one or more electronic documents;
for each portion of the media content of the portions of media content:
identifying a document portion, from the one or more electronic documents, that corresponds to the portion of media content;
generating an annotation that represents the portion of media content and associating the annotation to a location corresponding to the document portion within a particular electronic document of the one or more electronic documents; and

displaying, in electronic form within a display window, the one or more electronic documents with their corresponding one or more generated annotations from the portions of media content.

2 . The apparatus of claim 1 , wherein the one or more media content items are at least one of an audio file, a video file, a captured screenshot, or an interactive whiteboard file that contains a series of coordinates corresponding to received input representing generated marks on an interactive whiteboard.

3 . The apparatus of claim 1 , the one or more memories store additional instructions which, when processed by the one or more processors, cause:
generating updated one or more electronic documents that each include their associated annotations corresponding to identified document portions.

4 . The apparatus of claim 1 , wherein generating the annotation that represents the portion of media content associating the annotation to the location corresponding to the document portion within the particular electronic document of the one or more electronic documents, comprises:
generating the annotation comprising a text transcription of the portion of media content and an electronic link to the media content item containing the portion of media content, wherein the electronic link to the media content item is queued to play the portion of media content; and associating the annotation to the location corresponding to the document portion within the particular electronic document of the one or more electronic documents.

5 . The apparatus of claim 1 , wherein for each portion of the media content of the identified portions of media content, further cause:
inserting meeting details into the annotation, wherein the meeting details include at least one of a meeting name, meeting place, and meeting date and time; and inserting a user ID into the annotation that identifies a speaking user within the portion of media content.

6 . The apparatus of claim 1 , wherein identifying the portions of media content corresponding to content suggestions for the one or more electronic documents, comprises:
using a machine-learning model, identifying the portions of media content that correspond to phrases indicating the content suggestions for the one or more electronic documents, wherein the machine-learning model has been trained using an input data set of media content items that have identified content suggestion speech.

7 . Th apparatus of claim 6 , the one or more memories store additional instructions which, when processed by the one or more processors, cause:
using the machine-learning model, determining a content suggestion type for each of the portions of media content, wherein the content suggestion type is one of a comment or a suggested edit; upon generating the annotation that represents the portion of media content, determining that the annotation for the portion of media content corresponds to a suggested edit to the document portion; calculating a confidence score for the suggested edit, wherein the confidence score represents a level of confidence that the portion of media content corresponds to the suggested edit to the document portion; determining that the confidence score for the suggested edit is above a confidence score threshold for automatically editing the document portion; and automatically editing the document portion to reflect changes proposed in the suggested edit.

8 . The apparatus of claim 1 , wherein identifying the document portion, from the one or more electronic documents, that corresponds to the portion of media content, comprises:
using a first machine-learning model, identifying document portions within the one or more electronic documents based upon a determined document types associated with the one or more electronic documents and combinations of words within the one or more electronic documents, wherein the first machine-learning model has been trained using a plurality of documents of different document types; and using a second machine-learning model, correlating the document portion of the document portions to the portion of media content based upon a relative position determined for the portion of media content and a text transcription of the portion of media content, wherein the second machine-learning model has been trained using a plurality of document portions from a plurality of electronic documents and corresponding content suggestions for the plurality of document portions from the plurality of electronic documents.

9 . One or more non-transitory computer-readable media storing instructions which, when processed by one or more processors, cause:
receiving one or more media content items representing captured content from a discussion of one or more electronic documents by one or more users; identifying, from the one or more media content items, portions of media content corresponding to content suggestions for the one or more electronic documents; for each portion of the media content of the portions of media content:
identifying a document portion, from the one or more electronic documents, that corresponds to the portion of media content;
generating an annotation that represents the portion of media content and associating the annotation to a location corresponding to the document portion within a particular electronic document of the one or more electronic documents; and
displaying, in electronic form within a display window, the one or more electronic documents with their corresponding one or more generated annotations from the portions of media content.

10 . The one or more non-transitory computer-readable media of claim 9 , wherein the one or more media content items are at least one of an audio file, a video file, a captured screenshot, or an interactive whiteboard file that contains a series of coordinates corresponding to received input representing generated marks on an interactive whiteboard.

11 . The one or more non-transitory computer-readable media of claim 9 , further comprising additional instructions which, when processed by the one or more processors, cause:
generating updated one or more electronic documents that each include their associated annotations corresponding to identified document portions.

12 . The one or more non-transitory computer-readable media of claim 9 , wherein generating the annotation that represents the portion of media content associating the annotation to the location corresponding to the document portion within the particular electronic document of the one or more electronic documents, comprises:
generating the annotation comprising a text transcription of the portion of media content and an electronic link to the media content item containing the portion

Assignees

• Nelson Steven A
• Kitada Hiroshi
• Wong Lana
• Ricoh Co Ltd

Inventors

• Nelson Steven A
• Kitada Hiroshi
• Wong Lana

Classifications

• G06F40/169

  Annotation, e.g. comment data or footnotes · CPC title

• G06N20/00

  Machine learning · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06F40/117 Primary

  Tagging; Marking up (details of markup languages G06F40/143); Designating a block; Setting of attributes (style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154) · CPC title

• G06Q10/101 Primary

  Collaborative creation, e.g. joint development of products or services · CPC title