Creative intent scalability via physiological monitoring

Patent metadata

Field	Value
Publication number	US-11678014-B2
Application number	US-202217930357-A
Country	US
Kind code	B2
Filing date	Sep 7, 2022
Priority date	Oct 1, 2018
Publication date	Jun 13, 2023
Grant date	Jun 13, 2023

Abstract

Creative intent input describing emotion expectations and narrative information relating to media content is received. Expected physiologically observable states relating to the media content are generated based on the creative intent input. An audiovisual content signal with the media content and media metadata comprising the physiologically observable states is provided to a playback apparatus. The audiovisual content signal causes the playback device to use physiological monitoring signals to determine, with respect to a viewer, assessed physiologically observable states relating to the media content and generate, based on the expected physiologically observable states and the assessed physiologically observable states, modified media content to be rendered to the viewer.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving an audiovisual content signal including game media content and media metadata, wherein the media metadata comprises metadata corresponding to one or more expected physiologically observable states for one or more portions of the game media content and wherein the one or more expected physiologically observable states relate to emotion expectations and narrative information corresponding to one or more portions of the game media content;
obtaining one or more physiological monitoring signals from a viewer of the game media content;
determining, with respect to the viewer, one or more assessed physiologically observable states relating to the one or more portions of the game media content;
generating and rendering, based at least in part on the one or more expected physiologically observable states and the one or more assessed physiologically observable states, modified game media content from the game media content; and
presenting the modified game media content to the viewer.

2. The method of claim 1 , wherein the one or more assessed physiologically observable states comprise an assessed emotional state of the viewer and wherein the one or more expected physiologically observable states comprise an expected emotional state of the viewer.

3. The method of claim 2 , wherein the assessed emotional state and the expected emotional state correspond to at least one of arousal or valence.

4. The method of claim 1 , wherein the one or more assessed physiologically observable states comprise an assessed narrative state of the viewer and wherein the one or more expected physiologically observable states comprise an expected narrative state of the viewer.

5. The method of claim 4 , wherein the expected narrative state and the assessed narrative state correspond to one or more of stress, cognitive load or attention locus.

6. The method of claim 4 , wherein the expected narrative state corresponds to an expected confusion index.

7. The method of claim 1 , wherein the one or more assessed physiologically observable states comprise an assessed attention locus of the viewer and wherein the one or more expected physiologically observable states comprise an expected attention locus of the viewer.

8. The method of claim 1 , wherein the media metadata comprises one or more modification options for modifying the one or more portions of the game media content in response to detecting a divergence between the one or more assessed physiologically observable states and the one or more expected physiologically observable states.

9. The method of claim 8 , wherein at least one modification of the one or more modification options comprises instructions for implementing a game media content modification involving one or more of: luminance, spatial resolution, contrast, color saturation, hue, tone mapping, field of view, color gamut, luminance dynamic range, bit depth, spatial filtering, image refresh rate, one or more regions of interest, one or more audio objects of interest, zoom-in or -out factors, image steering, nonvisual characteristics, motion rendering characteristics, pivots, slopes and offsets of luminance mappings, luminance distribution, luminance in specific image regions, specific objects, specific characters, background, positions of audio objects, frequency equalization, reverberation, timbre, phase, number of speakers, speaker configuration, frequency ranges of speakers, phase distortions of speakers, loudspeaker selection, volume, actual audio channel configuration, snap tolerance options for selecting single speaker rendering and for selecting multi-speaker interpolation, one or more audio object positions, one or more audio object sizes, audio object radii, one or more audio object directions, or one or more audio object trajectories, dialog volume, non-dialog volume, dialog enhancement, audio dynamic range, specific loudspeaker selection, specific loudspeaker configuration, echo characteristics, delays, signal attack times, or signal release times.

10. The method of claim 8 , wherein at least one modification of the one or more modification options comprises instructions for implementing an image steering modification and wherein the image steering modification involves steering images to follow the viewer's movements from room to room.

11. The method of claim 8 , wherein at least one modification of the one or more modification options comprises instructions for implementing an attention steering modification and wherein the attention steering modification involves steering the viewer's attention locus towards an area of interest of the game media content, towards a region of interest of the game media content, away from an area of interest of the game media content, or away from a region of interest of the game media content.

12. The method of claim 8 , wherein at least one modification of the one or more modification options comprises instructions for implementing a game media content modification involving one or more of: one or more visual characteristics of a sequence of rendered images, one or more visual characteristics of a visual scene bounded by two consecutive scene cuts, one or more visual characteristics of a subdivision of a visual scene, one or more visual characteristics of a group of pictures (GOP), one or more visual characteristics of one or more tile-sized regions spanning multiple frames, one or more visual characteristics of portions of a spatiotemporal stream, one or more visual characteristics of an entire image, one or more visual characteristics of an image region that depicts a specific character or one or more visual characteristics of an image region that depicts a specific object.

13. The method of claim 8 , wherein the one or more modification options are used to minimize the divergence between the one or more assessed physiologically observable states and the one or more expected physiologically observable states, with respect to the viewer, in content playback of the game media content.

14. The method of claim 8 , wherein the one or more physiological monitoring signals are obtained from one or more of: display-based sensors, visible wavelength camera sensors, simultaneous localization and mapping sensors, thermal imagers, head-mounted-display sensors, in-ear sensors, wrist sensors, gaze position sensors, pupil diameter sensors, facial expression sensors, head position sensors, viewing distance sensors, facial expression sensors, valence sensors, arousal sensors, electroencephalogram sensors, specifically positioned electrodes, thermal sensors, optical sensors, electro-oculogram sensors, respiration sensors, plethysmography-heartrate-based sensors, galvanic skin response sensors, gas sensors, CO2 content sensors, R3COH content sensors, or seat-based sensors.

15. The method of claim 8 , wherein the one or more signal modification options are generated based at least in part on playback device characterization data, rendering environment characterization data, or a combination thereof.

16. An apparatus, comprising:
an interface system; and
a control system configured to:
receive, via the interface system, an audiovisual content signal including game media content and media metadata, wherein the media metadata comprises metadata corresponding to one or more expected physiologically observable states for one or more portions of the game media content and wherein the one or more expected physiologically observable states relate to emotion expectations and narrative information corresponding to one or more portions

Assignees

• Dolby Laboratories Licensing Corp

Inventors

• Daly Scott
• Crum Poppy Anne Carrie
• Gitterman Evan David
• Ruggieri Shane Mario

Classifications

• H04N21/41415

  involving a public display, viewable by several users in a public space outside their home, e.g. movie theatre, information kiosk · CPC title

• H04N21/854

  Content authoring · CPC title

• H04N21/440245

  the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment · CPC title

• H04N21/44218 Primary

  Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45) · CPC title

• H04N21/84 Primary

  Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)} · CPC title