Creative intent scalability via physiological monitoring

Patent metadata

Field	Value
Publication number	US-11477525-B2
Application number	US-201917281946-A
Country	US
Kind code	B2
Filing date	Sep 30, 2019
Priority date	Oct 1, 2018
Publication date	Oct 18, 2022
Grant date	Oct 18, 2022

Abstract

Creative intent input describing emotion expectations and narrative information relating to media content is received. Expected physiologically observable states relating to the media content are generated based on the creative intent input. An audiovisual content signal with the media content and media metadata comprising the physiologically observable states is provided to a playback apparatus. The audiovisual content signal causes the playback device to use physiological monitoring signals to determine, with respect to a viewer, assessed physiologically observable states relating to the media content and generate, based on the expected physiologically observable states and the assessed physiologically observable states, modified media content to be rendered to the viewer.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving creative intent input describing emotion expectations and narrative information relating to one or more portions of media content;
generating, based at least in part on the creative intent input, one or more expected physiologically observable states relating to the one or more portions of the media content;
providing, to a playback device, an audiovisual content signal with the media content and media metadata comprising the one or more expected physiologically observable states for the one or more portions of the media content;
wherein the audiovisual content signal causes the playback device (a) to use one or more physiological monitoring signals to determine, with respect to a viewer, one or more assessed physiologically observable states relating to the one or more portions of the media content and (b) to generate, based at least in part on the one or more expected physiologically observable states and the one or more assessed physiologically observable states, modified media content from the media content as the modified media content generated from the media content is being adjusted and rendered to the viewer.

2. The method of claim 1 , wherein the creative intent input represents creative intent of creatives who cause the media content and the media metadata to be generated in a production stage.

3. The method of claim 1 , wherein the creative intent input contains semantic expressions of creatives' intent, wherein the media metadata comprises one of: the semantic expressions used to derive a set of non-semantic signal modification options in a consumption stage or the set of non-semantic signal modification options generated based on the semantic expressions in a production stage, and wherein the playback device selects one or more specific signal modification options from the set of signal modification options to perform one or more media content adjustments to the media content to minimize a divergence the one or more expected physiologically observable states and the one or more assessed physiologically observable states in response to determining that the divergence is greater than a divergence threshold.

4. A computer-implemented method comprising:
receiving an audiovisual content signal with media content and media metadata, wherein the media metadata comprises one or more expected physiologically observable states for one or more portions of the media content;
wherein the one or more expected physiologically observable states relating to the one or more portions of the media content are generated based at least in part on creative intent input describing emotion expectations and narrative information relating to one or more portions of media content;
using one or more physiological monitoring signals to determine, with respect to a viewer, one or more assessed physiologically observable states relating to the one or more portions of the media content;
generating and rendering, based at least in part on the one or more expected physiologically observable states and the one or more assessed physiologically observable states, modified media content from the media content as the modified media content generated from the media content is being adjusted and rendered to the viewer.

5. The method of claim 4 , wherein the one or more assessed physiologically observable states comprise an assessed emotional state of the viewer, wherein the one or more expected physiologically observable states comprise an expected emotional state, of the viewer, that is of a same emotional state type as the assessed emotional state of the viewer.

6. The method of claim 4 , wherein the one or more assessed physiologically observable states comprise an assessed narrative state of the viewer, wherein the one or more expected physiologically observable states comprise an expected narrative state, of the viewer, that is of a same narrative state type as the assessed narrative state of the viewer.

7. The method of claim 4 , wherein the one or more assessed physiologically observable states comprise an assessed attention locus of the viewer, wherein the one or more expected physiologically observable states comprise an expected attention locus of the viewer.

8. The method of claim 4 , wherein the media metadata comprises one or more signal modification options for modifying the one or more portions of the media content in response to detecting a divergence between the one or more assessed physiologically observable states and the one or more expected physiologically observable states.

9. The method of claim 8 , wherein at least one signal modification of the one or more signal modification options comprises instructions for implementing a media content modification on one of more of: luminance, spatial resolution, contrast, color saturation, hue, tone mapping, field of view, color gamut, luminance dynamic range, bit depth, spatial filtering, image refresh rate, zoom-in or -out factors, image steering, non-visual characteristics, motion rendering characteristics, pivots, slopes and offsets of luminance mappings, luminance distribution, luminance in specific image regions, specific objects, specific characters, background, positions of audio objects, frequency equalization, reverberation, timbre, phase, number of speakers, speaker configuration, frequency ranges of speakers, phase distortions of speakers, loudspeaker selection, volume, actual audio channel configuration, snap tolerance options for selecting single speaker rendering and for selecting multi-speaker interpolation, audio object positions, audio object sizes, audio object radii, audio object directions, audio object trajectories, dialog volume, non-dialog volume, dialog enhancement, audio dynamic range, specific loudspeaker selection, specific loudspeaker configuration, echo characteristics, delays, signal attack times, or signal release times.

10. The method of claim 8 , wherein the one or more signal modification options are used to minimize the divergence between the one or more assessed physiologically observable states and the one or more expected physiologically observable states, with respect to the viewer, in content playback of the media content.

11. The method of claim 8 , wherein the one or more physiological monitoring signals are generated by one or more of: display-based sensors, visible wavelength camera sensors, simultaneous localization and mapping sensors, thermal imagers, head-mounted-display sensors, in-ear sensors, wrist sensors, gaze position sensors, pupil diameter sensors, facial expression sensors, head position sensors, viewing distance sensors, facial expression sensors, valence sensors, arousal sensors, electroencephalogram sensors, specifically positioned electrodes, thermal sensors, optical sensors, electro-oculogram sensors, respiration sensors, plethysmography-heartrate-based sensors, galvanic skin response sensors, gas sensors, CO2 content sensors, R3COH content sensors, or seat-based sensors.

12. The method of claim 8 , wherein the one or more signal modification options are generated based at least in part on playback device characterization data and rendering environment characterization data.

13. A system performing any of the method recited in claim 1 .

14. A non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors cause performance of the method recited in claim 1 .

15. An apparatus comprising one or more processors and one or more storage media, storing a set of instructions, which when executed by one or m

Assignees

• Dolby Laboratories Licensing Corp

Inventors

• Daly Scott
• Crum Poppy Anne Carrie
• Gitterman Evan David
• Ruggieri Shane Mario

Classifications

• H04N21/4126 Primary

  The peripheral being portable, e.g. PDAs or mobile phones · CPC title

• H04N21/4621

  Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen (arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10) · CPC title

• H04N21/44218 Primary

  Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45) · CPC title

• H04N21/84 Primary

  Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)} · CPC title

• H04N21/4223

  Cameras (H04N23/00 takes precedence) · CPC title