Personalized sensory feedback

Patent metadata

Field	Value
Publication number	US-11929169-B2
Application number	US-202217650438-A
Country	US
Kind code	B2
Filing date	Feb 9, 2022
Priority date	Feb 9, 2022
Publication date	Mar 12, 2024
Grant date	Mar 12, 2024

Abstract

Personalizing sensory feedback based on user sensitivity analysis includes maintaining user-specific parameters for provision of sensory feedback to a user in extended reality. The user-specific parameters apply to specific contextual situations and dictate levels of sensory feedback to provide via stimulus device(s) in the specific contextual situations. Based on an ascertained contextual situation of the user interacting in a target extended reality environment, a set of sensory feedback level parameters is selected for provision of sensory feedback to the user in the target extended reality environment, and stimulus device(s) in the target extended reality environment is/are automatically controlled in the provision of the sensory feedback to the user based on one or more of the selected parameters. The automatically controlling includes electronically communicating with the stimulus device(s) to control at least one stimuli provided to the user by the stimulus device(s).

First claim

What is claimed is:

1. A computer-implemented method comprising:
maintaining user-specific parameters for provision of sensory feedback to a user in extended reality, the user-specific parameters applying to specific contextual situations and dictating levels of sensory feedback to provide via one or more stimulus devices in the specific contextual situations, wherein the maintaining comprises using feedback captured from user responses to generated prompts to the user, as input to train a Multi-Agent Reinforcement Learning (MARL) artificial intelligence (AI) model to identify the user-specific parameters applying to the specific contextual situations;
based on an ascertained contextual situation of the user interacting in a target extended reality environment, selecting a set of sensory feedback level parameters for provision of sensory feedback to the user in the target extended reality environment, wherein the sensory feedback is in response to generating a plurality of questions about a user's comfort level, wherein the selecting comprises applying the MARL AI model to features of the ascertained contextual situation and obtaining as an output of the MARL AI model a classification of the set of sensory feedback level parameters, wherein the sensory feedback is personalized to the user based on employing a reward function parametrized on factors including sentiment data and accelerometer data, wherein the sentiment data is detected by an NLP engine and the accelerometer data including the user's movement and orientation in space is captured with an accelerometer;
training the MARL AI model in real-time using the sensory feedback from a user apparatus producing the accelerometer data, the sensory feedback being received from the user in real-time including new reactions, iteratively adjusting the set of sensory feedback level parameters of the MARL AI model based on the sensory feedback; and
automatically controlling, in the provision of the sensory feedback to the user in the target extended reality environment, at least one stimulus device in the target extended reality environment based on one or more of the selected parameters, the automatically controlling comprising electronically communicating with the at least one stimulus device to control one or more stimuli provided to the user by the at least one stimulus device.

2. The method of claim 1 , wherein the maintaining the user-specific parameters comprises maintaining unique sets of sensory feedback level parameters correlating to different contextual situations in which user reactions have been previously observed.

3. The method of claim 2 , wherein the selecting comprises comparing the ascertained contextual situation to one or more of the different contextual situations and determining based on the comparing whether the ascertained contextual situation corresponds to a contextual situation of the different contextual situations, and wherein the selecting of the selected set of sensory feedback level parameters is based on the determining.

4. The method of claim 3 , wherein the determining determines that the ascertained contextual situation corresponds to a contextual situation of the different contextual situations, and wherein the selected set of sensory feedback level parameters is the unique set of sensory feedback level parameters correlating to the contextual situation to which the ascertained contextual situation corresponds.

5. The method of claim 3 , wherein maintaining the user-specific parameters comprises maintaining one or more sets of anticipated preferred sensory feedback levels for the user, wherein the determining determines that the ascertained contextual situation does not correspond to any contextual situation of the different contextual situations, and wherein the selected set of sensory feedback level parameters is a set of anticipated preferred sensory feedback level parameters of the one or more sets of anticipated preferred sensory feedback levels for the user.

6. The method of claim 5 , wherein the selected set of anticipated preferred sensory feedback level parameters for the user is based on a template set of sensory feedback level parameters built based on a population of users in which the user is classified.

7. The method of claim 1 , wherein the maintaining further comprises using feedback employing observed user reactions to provided stimuli in one or more extended reality environments.

8. The method of claim 1 , wherein different specific contextual situations comprises different extended reality environments comprising different sensors and stimulus devices thereof.

9. The method of claim 1 , wherein the target extended reality environment comprises at least one selected from the group consisting of a virtual reality environment, an augmented reality environment, and a mixed-reality environment.

10. A computer system comprising:
a memory; and
a processor in communication with the memory, wherein the computer system is configured to perform a method comprising:
maintaining user-specific parameters for provision of sensory feedback to a user in extended reality, the user-specific parameters applying to specific contextual situations and dictating levels of sensory feedback to provide via one or more stimulus devices in the specific contextual situations, wherein the maintaining comprises using feedback captured from user responses to generated prompts to the user, as input to train a Multi-Agent Reinforcement Learning (MARL) artificial intelligence (AI) model to identify the user-specific parameters applying to the specific contextual situations;
based on an ascertained contextual situation of the user interacting in a target extended reality environment, selecting a set of sensory feedback level parameters for provision of sensory feedback to the user in the target extended reality environment, wherein the sensory feedback is in response to generating a plurality of questions about a user's comfort level, wherein the selecting comprises applying the MARL AI model to features of the ascertained contextual situation and obtaining as an output of the MARL AI model a classification of the set of sensory feedback level parameters, wherein the sensory feedback is personalized to the user based on employing a reward function parametrized on factors including sentiment data and accelerometer data, wherein the sentiment data is detected by an NLP engine and the accelerometer data including the user's movement and orientation in space is captured with an accelerometer;
training the MARL AI model in real-time using the sensory feedback from a user apparatus producing the accelerometer data, the sensory feedback being received from the user in real-time including new reactions, iteratively adjusting the set of sensory feedback level parameters of the MARL AI model based on the sensory feedback; and
automatically controlling, in the provision of the sensory feedback to the user in the target extended reality environment, at least one stimulus device in the target extended reality environment based on one or more of the selected parameters, the automatically controlling comprising electronically communicating with the at least one stimulus device to control one or more stimuli provided to the user by the at least one stimulus device.

11. The computer system of claim 10 , wherein the maintaining the user-specific parameters comprises maintaining unique sets of sensory feedback level parameters correlating to different contextual situations in which user reactions have been previously observed.

12. The computer system of claim 11 , wherein the selecting comprises comparing the ascertained contextual situati

Assignees

• Kyndryl Inc

Inventors

• Miles Caleb
• Kwatra Shikhar
• Szkatulski Jennifer L
• Sanabria Echeverria Elio Andres

Classifications

• G16H40/67 Primary

  for remote operation · CPC title

• G06N20/00

  Machine learning · CPC title

• G06N3/006 Primary

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title

• G06N3/092

  Reinforcement learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title