Systems and methods for determining a type of material of an object in a real-world environment

Patent metadata

Field	Value
Publication number	US-12318684-B2
Application number	US-202217589854-A
Country	US
Kind code	B2
Filing date	Jan 31, 2022
Priority date	Jan 31, 2022
Publication date	Jun 3, 2025
Grant date	Jun 3, 2025

Abstract

Methods and systems for determining a type of material of an object in a real-world environment are described. One of the methods includes receiving a plurality of sets of audio data based on sounds received from a plurality of objects within a plurality of environments. The method further includes receiving a plurality of sets of input data regarding a plurality of types of materials of the plurality of objects, training an artificial intelligence (AI) model based on the plurality of sets of audio data and the plurality of sets of input data, and applying the AI model to a set of audio data captured from the real-world environment to determine the type of material of the object within the real-world environment to enhance the realism of augmented reality (AR) or virtual reality (VR) video games and applications.

First claim

The invention claimed is:

1. A method for determining a type of material of a real-world object in a real-world environment, comprising:
capturing, by a microphone, a set of audio data based on sounds output in the real-world environment, wherein the real-world environment includes the real-world object, wherein the set of audio data is in a digital format;
providing the set of audio data captured from the real-world environment to an artificial intelligence (AI) model for determining the type of material of the real-world object within the real-world environment, wherein the set of audio data is processed to determine an audio parameter of the set of audio data and the type of material is determined based on a comparison of the audio parameter of the set of audio data with an additional audio parameter of an additional audio data set;
rendering, in a virtual environment, a first virtual object having the type of material in of the real-world object; and
providing the set of audio data to be output as sound when the first virtual object in the virtual environment interacts with a second virtual object in the virtual environment.

2. The method of claim 1 , further comprising:
receiving a plurality of sets of audio data based on sounds received from a plurality of real-world objects within a plurality of real-world environments;
additionally receiving a plurality of output parameters regarding the plurality of real-world objects within the plurality of real-world environments;
training the AI model based on the plurality of sets of audio data and the plurality of output parameters.

3. The method of claim 2 , further comprising extracting a plurality of features from the plurality of sets of audio data, wherein the plurality of features include a plurality of amplitudes of the plurality of sets of audio data and a plurality of frequencies of the plurality of sets of audio data, wherein the plurality of sets of audio data include the additional audio data set and the plurality of features include the additional audio parameter.

4. The method of claim 3 , further comprising:
classifying the plurality of features to output associations between the plurality of features and the plurality of output parameters,
wherein said training the AI model includes providing, to the AI model, the associations between the plurality of features and the plurality of output parameters,
wherein said determining the type of material includes determining, by the AI model, based on the associations between the plurality of features and the plurality of output parameters, a probability that the set of audio data captured from the real-world environment indicates the type of material of the real-world object.

5. The method of claim 3 , further comprising receiving a plurality of sets of input data including a plurality of sets of image data captured by a plurality of cameras in the plurality of real-world environments, or light detection and ranging (LiDAR) data captured from the plurality of real-world environments, or inertial measurement unit (IMU) data captured from the plurality of real-world environments, or a combination thereof, wherein said extracting the plurality of features comprises extracting the plurality of features from the plurality of sets of input data, wherein the plurality of features include a plurality of identities of the plurality of real-world objects, a plurality of physical parameters defining movements of the plurality of real-world objects and a plurality of graphical parameters of the plurality of real-world objects.

6. The method of claim 5 , wherein the plurality of physical parameters include a first physical parameter and a second physical parameter, wherein the plurality of graphical parameters include a first graphical parameter and a second graphical parameter, the method further comprising:
classifying the plurality of features to output associations between the plurality of features and the plurality of output parameters,
wherein said training the AI model includes providing, to the AI model, the associations between the plurality of features and the plurality of output parameters.

7. The method of claim 6 , wherein said determining the type of material of the real-world object within the real-world environment includes determining a probability that the real-world object has the type of material, wherein the probability is determined based on the associations between the plurality of features and the plurality of output parameters.

8. The method of claim 7 , further comprising:
generating virtual object data for display of the first virtual object as interacting with the second virtual object in the virtual environment;
accessing the set of audio data to be output by the first virtual object as the first virtual object interacts with the second virtual object; and
generating haptic feedback data to be output by a body connected device as the first virtual object interacts with the second virtual object, wherein the virtual object data represents the type of material, a movement of the real-world object from the first physical parameter to the second physical parameter in the real-world environment, and a change from the first graphical parameter to the second graphical parameter of the real-world object in the real-world environment.

9. The method of claim 2 , wherein the plurality of output parameters include data identifying the plurality of materials of the plurality of real-world objects in the plurality of real-world environments, the method further comprising simulating a virtual interaction between the first virtual object of a video game and the second virtual object based on the type of material.

10. The method of claim 2 , further comprising receiving a plurality of sets of input data, wherein the plurality of sets of input data include a plurality of sets of image data captured by a plurality of cameras in the plurality of real-world environments, wherein the plurality of sets of audio data and the plurality of image data sets are captured when the plurality of real-world objects interact with a plurality of users in the plurality of real-world environments.

11. A server for determining a type of material of a real-world object in a real-world environment, comprising:
a processor configured to:
receive, from a microphone in the real-world environment, a set of audio data, wherein the set of audio data has a digital format and is captured based on sounds output in the real-world environment, wherein the real-world environment includes the real-world object;
provide the set of audio data captured from the real-world environment to an artificial intelligence (AI) model to determine the type of material of the real-world object within the real-world environment, wherein the set of audio data is provided to be processed, wherein the set of audio data is processed to determine an audio parameter of the set of audio data and the type of material is determined based on a comparison of the audio parameter of the set of audio data with an additional audio parameter of an additional audio data set;
provide the type of material to a client device to facilitate a display, in a virtual environment, of a first virtual object having the type of material of the real-world object; and
provide the set of audio data to be output as sound when the first virtual object interacts with a second virtual object in the virtual environment; and

a memory device coupled to the processor.

12. The server of claim 11 , wherein the processor is configured to:
receive a plurality of sets of audio data based on sounds received from a plurality of real-world objects within

Assignees

• Sony Interactive Entertainment Inc

Inventors

• Stafford Jeffrey
• Young Andrew
• Tokubo Todd

Classifications

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B27/0101

  characterised by optical features (G02B27/0172 takes precedence) · CPC title

• A63F13/217

  using environment-related information, i.e. information generated otherwise than by the player, e.g. ambient temperature or humidity · CPC title

• A63F13/57

  Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game (A63F13/56 takes precedence) · CPC title

• A63F13/211

  using inertial sensors, e.g. accelerometers or gyroscopes · CPC title