Device and method for generating summary video

Patent metadata

Field	Value
Publication number	US-2022012500-A1
Application number	US-202117372276-A
Country	US
Kind code	A1
Filing date	Jul 9, 2021
Priority date	Jul 9, 2020
Publication date	Jan 13, 2022
Grant date	—

Abstract

A method for generating a summary video includes generating a user emotion graph of a user watching a first video. The method also includes obtaining a character emotion graph for a second video, by analyzing an emotion of a character in a second video that is a target of summarization. The method further includes obtaining an object emotion graph for an object in the second video, based on an object appearing in the second video. Additionally the method includes obtaining an image emotion graph for the second video, based on the character emotion graph and the object emotion graph. The method also includes selecting at least one first scene in the second video by comparing the user emotion graph with the image emotion graph. The method further includes generating the summary video of the second video, based on the at least one first scene.

First claim

What is claimed is:

1 . A method, performed by a device, of generating a summary video, the method comprising:
obtaining a user image in which a user watching a first video is photographed, during playback of the first video; generating a user emotion graph of the user watching the first video, by analyzing an emotion of the user in the obtained user image; obtaining a character emotion graph for a second video, by analyzing an emotion of a character in the second video that is a target of summarization; obtaining an object emotion graph for an object in the second video, based on the object appearing in the second video; obtaining an image emotion graph for the second video, based on the character emotion graph and the object emotion graph; selecting at least one first scene in the second video by comparing the user emotion graph of the user that watched the first video with the image emotion graph for the second video; and generating the summary video of the second video, based on the at least one first scene.

2 . The method of claim 1 , further comprising:
selecting at least one second scene in the second video, based on emotion scores in the image emotion graph, wherein the generating of the summary video comprises generating the summary video of the second video, based on the at least one first scene and the at least one second scene.

3 . The method of claim 2 , wherein selecting the at least one first scene comprises:
selecting a first part of the image emotion graph by comparing a slope of emotion curves in the user emotion graph with a slope of emotion curves in the image emotion graph, and selecting a scene corresponding to the selected first part as the at least one first scene to be included in the summary video.

4 . The method of claim 2 , wherein selecting the at least one second scene comprises:
selecting a second part comprising an emotion score greater than a certain threshold value when an emotion score of at least one preset emotion in the image emotion graph is greater than the certain threshold value, and selecting a scene corresponding to the selected second part as the at least one second scene to be included in the summary video.

5 . The method of claim 2 , wherein generating the summary video comprises combining frames corresponding to the at least one first scene with frames corresponding to the at least one second scene.

6 . The method of claim 1 , wherein the user emotion graph is generated based on a user emotion score calculated based on at least one of a facial expression or a voice of the user watching the first video.

7 . The method of claim 1 , wherein:
the character emotion graph is generated based on a character emotion score calculated for a frame or a scene of the second video, and obtaining the character emotion graph comprises calculating the character emotion score for a frame or a scene of the second video, by applying different weight values to characters in the second video.

8 . The method of claim 1 , wherein obtaining the object emotion graph comprises:
obtaining previously generated object emotion information based on at least one of a facial expression, a voice, or a line of a character at a time of appearance of the object in at least one third video; calculating an object emotion score for the object appearing in the second video for a frame or a scene of the second video, based on the obtained object emotion information; and generating an object emotion graph for the second video, based on the calculated object emotion score.

9 . The method of claim 1 , further comprising:
identifying a sound for a frame or a scene in the second video; and obtaining a sound emotion graph for a sound output from the second video, based on the identified sound, wherein obtaining the image emotion graph comprises obtaining the image emotion graph for the second video, based on the character emotion graph, the object emotion graph, and the sound emotion graph.

10 . The method of claim 9 , further comprising:
identifying a line for a frame or a scene in the second video; and obtaining a line emotion graph for a line output from the second video, based on the identified line, wherein obtaining the image emotion graph comprises obtaining the image emotion graph for the second video, based on the character emotion graph, the object emotion graph, the sound emotion graph, and the line emotion graph.

11 . A device for generating a summary video, the device comprising:
a communication interface; a display; a memory storing one or more instructions; and a processor configured to:
obtain a user image in which a user watching a first video is photographed, during playback of the first video through the display,
generate a user emotion graph of the user watching the first video, by analyzing an emotion of the user in the obtained user image,
obtain a character emotion graph for a second video, by analyzing an emotion of a character in the second video that is a target of summarization,
obtain an object emotion graph for an object in the second video, based on an object appearing in the second video,
obtain an image emotion graph for the second video, based on the character emotion graph and the object emotion graph,
select at least one first scene in the second video by comparing the user emotion graph of the user that watched the first video with the image emotion graph for the second video, and
generate the summary video of the second video, based on the at least one first scene.

12 . The device of claim 11 , wherein the processor is further configured to:
select at least one second scene in the second video based on emotion scores in the image emotion graph; and generate the summary video of the second video based on the at least one first scene and the at least one second scene.

13 . The device of claim 12 , wherein the processor is further configured to:
select a first part of the image emotion graph by comparing a slope of emotion curves in the user emotion graph with a slope of emotion curves in the image emotion graph; and select a scene corresponding to the selected first part as the at least one first scene to be included in the summary video.

14 . The device of claim 12 , wherein the processor is further configured to:
select a second part including an emotion score greater than a certain threshold value when an emotion score of at least one preset emotion in the image emotion graph is greater than the certain threshold value; and select a scene corresponding to the selected second part as the at least one second scene to be included in the summary video.

15 . The device of claim 12 , wherein to generate the summary video, the processor is further configured to combine frames corresponding to the at least one first scene with frames corresponding to the at least one second scene.

16 . The device of claim 11 , wherein the user emotion graph is generated based on a user emotion score calculated based on at least one of a facial expression or a voice of the user watching the first video.

17 . The device of claim 11 , wherein:
the character emotion graph is generated based on a character emotion score calculated for a frame or a scene of the second video, and the processor is further configured to: calculate the character emotion score for a frame or a scene of the second video, by applying different weight values to characters in the second video.

18 . The device of claim 11 , wherein the processor is further configured to:
o

Assignees

• Samsung Electronics Co Ltd

Inventors

• Han Junkyu
• Lee Pius
• Tak Hyunuk

Classifications

• G10L25/63

  for estimating an emotional state · CPC title

• H04N21/4667

  Processing of monitored end-user data, e.g. trend analysis based on the log file of viewer selections · CPC title

• H04N21/8549 Primary

  Creating video summaries, e.g. movie trailer {(retrieval in video databases by using presentations in form of a video summary G06F16/739)} · CPC title

• G06V20/47 Primary

  Detecting features for summarising video content · CPC title

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title