Systems and methods for emotion and perception based video compression and video perception enhancement

Patent metadata

Field	Value
Publication number	US-11483574-B2
Application number	US-202017134845-A
Country	US
Kind code	B2
Filing date	Dec 28, 2020
Priority date	Dec 26, 2019
Publication date	Oct 25, 2022
Grant date	Oct 25, 2022

Abstract

A computer-implemented system and associated methods are disclosed including a processor and a camera. The processor is adapted to map emotional responses to video characteristics in view of reaction data as captured from the camera as the camera records being displayed a plurality of video data streams and also records the user response in view of modifications to the plurality of data streams. The map is implemented by the processor to generate revised or altered versions of the underlying video data streams for predetermined applications. Portions of the revised versions of the plurality of video data streams determined by the map to trigger an emotional response may be reduced, minimized, compressed, enhanced, altered, or otherwise modified in view of the corresponding emotional response as desired.

First claim

What is claimed is:

1. A computer-implemented system for automated modification of video data according to desired emotional characteristics, comprising:
a codec that encodes and decodes data from a plurality of video data streams;
a camera that captures reaction data of a user viewing the plurality of video data streams via a display; and
a processor associated with the codec that accesses the reaction data from the camera and the plurality of video data streams, the processor adapted to:
detect an emotional expression from the reaction data and a feature of the plurality of video data streams that elicits the emotional expression,
generate a map of video properties to emotional responses by the user from the reaction data, the emotional expression being mapped to the feature of the plurality of video data streams,
refine the map to define a selective modification to the feature and corresponding image frames of the plurality of video data streams for selective emotional enhancement of the emotional expression,
modify the plurality of video data streams in view of the map as refined to generate a revised plurality of video data streams with the feature and corresponding image frames of the plurality of video data streams selectively modified by the map as refined, and
execute an instruction to display to the user the revised plurality of video data streams with the feature as modified by the map to elicit the selective emotional enhancement of the emotional expression.

2. The computer-implemented system of claim 1 , wherein the processor is configured to continuously alter the feature in view of a new plurality of video data streams that include the feature to continuously elicit a change in the emotional expression.

3. The computer-implemented system of claim 2 , wherein the feature corresponds to one or more image frames of the plurality of video data streams determined by the processor to elicit the emotional expression from the user.

4. The computer-implemented system of claim 2 , wherein the change in the emotional expression is a removal, reduction, or enhancement of the emotional expression.

5. The computer-implemented system of claim 1 , wherein the emotional expression is identified by the processor from reaction image frames of the reaction data, the reaction image frames defining emotional characteristics including predetermined movements, predetermined facial expressions, or predetermined changes in focus areas of an image.

6. The computer-implemented system of claim 1 , wherein the feature is modified by adjusting image tonality, adjusting a resolution, or adjusting a field of view associated with one or more image frames of the plurality of video data streams.

7. The computer-implemented system of claim 1 , wherein the processor generates the map by executing a machine learning module that, when fed continuously with the plurality of video data streams and the reaction data, probes the reaction data for changes in the emotional expression as visual components of the reaction data change over time, and identifies as features one or more corresponding frames of the plurality of video data streams proximate to periods of the time during which the visual components of the reaction data change over time.

8. The computer-implemented system of claim 7 , wherein the machine learning module utilizes initial regression in view of a continuous video tuning loop from the plurality of video data streams and the reaction data.

9. The computer-implemented system of claim 7 , wherein the processor, via the machine learning module, analyzes the plurality of video data streams and the reaction data in parallel and identifies the feature of the plurality of video data streams by identifying a range of time during which the emotional expression is observed in the reaction data, and suggesting the feature is defined by one or more frames of the plurality of video data streams observable just prior to the range of time and the emotional expression observed in the reaction data.

10. The computer-implemented system of claim 1 , wherein the processor applies changes to new video data streams using the map in real-time by detecting a presence of the feature within the new video data streams and modifying the feature according to parameters of the map.

11. The system of claim 1 , wherein the feature of the plurality of video data streams is defined by an area of interest associated with the emotional expression, and the processor is further adapted to:
apply selective localized compression to portions of the plurality of video data streams outside the area of interest.

12. A method for selective modification of video properties according to a predetermined emotional characteristic objective, comprising:
accessing a video data stream by a processor;
generating, by the processor, a map of video properties to emotional responses from the video data stream, including:
displaying the video data stream to a user,
detecting an emotional feature associated with an emotional expression from reaction data captured by a camera and accessed by the processor as the user views the video data stream, and
detecting a video feature related to the emotional feature and further corresponding to one or more image frames of the video data stream; and
modifying, selectively by the processor, the one or more image frames for selective emotional tuning of the emotional expression.

13. A method of signal compression and alteration based on predetermined content, comprising:
accessing signal data by a processor;
analyzing the signal data, by the processor, to search for portions of the signal data having areas of interest related to predefined characteristics;
defining, by the processor, certain portions of the signal data as being associated with the areas of interest; and
applying, by the processor, compression to portions of the signal data devoid of the areas of interest to reduce resolution.

14. The method of claim 13 , wherein the predefined characteristics include predetermined movements, predetermined facial expressions, predetermined portions of audio, and predetermined focus areas of an image.

Assignees

• Coza Aurel
• Univ Arizona State

Inventors

• Coza Aurel

Classifications

• H04N19/162 Primary

  User input · CPC title

• G06V40/174

  Facial expression recognition · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• H04N19/117 Primary

  Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635) · CPC title

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title