Sound localization in an augmented reality view of a live event held in a real-world venue

Patent metadata

Field	Value
Publication number	US-2019105568-A1
Application number	US-201715730403-A
Country	US
Kind code	A1
Filing date	Oct 11, 2017
Priority date	Oct 11, 2017
Publication date	Apr 11, 2019
Grant date	—

Abstract

A method for localizing sound when enabling participation in a live event, including establishing a multi-player gaming session of a gaming application that generates an interactive gaming world, the live event being a real-world venue where players playing the gaming application are present. A live view of the venue is generated based on captured video streams, and generated for a physical POV anchored to a physical location in the venue. Audio is generated for the live event, and modified to reflect acoustics of the real-world venue for the physical location. The modified audio is synchronized to the live view. The live view and synchronized audio are streamed to an HMD of a remote user located outside the venue, and presents an augmented reality view of the live event.

First claim

What is claimed is:

1 . A method for localizing sound when enabling participation in a live event, comprising:
establishing a multi-player gaming session controlled by a plurality of players through execution of a gaming application at a server, the multi-player gaming session generating an interactive gaming world within which player participation is enabled, the live event being a real-world venue where the plurality of players is present; generating at the server a three dimensional (3D) live view of the real-world venue based on one or more captured video streams, the 3D live view generated for a first physical point-of-view (POV) of the live event, wherein the first physical POV is anchored to a first physical location in the real-world venue; generating audio of the live event based on one or more captured audio streams of the live event, the audio being broadcasted over a plurality of sound sources arranged throughout the real-world venue; modifying the audio to reflect acoustics of the real-world venue for the first physical location; synchronizing the audio that is modified to the 3D live view; and streaming the live view and audio that is synchronized and modified via a network to a head mounted display (HMD) of a first remote user for display, the first remote user being located outside of the real-world venue, the live view and audio that are streamed presenting a first augmented reality view of the live event to the first remote user through the HMD.

2 . The method of claim 1 , wherein the modifying further comprises:
performing a calibration process to determine sound characteristics of the real-world venue for the first physical location and for a plurality of locations of the sound sources in the real-world venue; generating acoustic modifications based on the sound characteristics; and applying the acoustic modifications when modifying the audio.

3 . The method of claim 2 , wherein the performing a calibration process comprises:
broadcasting an audio test pattern from the plurality of sound sources in the real-world venue; measuring audio signals received from the audio test pattern that is broadcasted at a plurality of test locations in the real-world venue; determining calibration measurements of the audio signals for each of the plurality of test locations; for the first physical location corresponding to the first physical POV in the real world venue, determining localized calibration measurements for the first physical location through interpolation of calibration measurements from a first set of test locations relevant to the first physical location; and generating the acoustic modifications for the first physical location based on the localized calibration measurements.

4 . The method of claim 2 , wherein the sound characteristics include timing characteristics, reflection characteristics, refraction characteristics, interference characteristics, and attenuation characteristics.

5 . The method of claim 1 , further comprising:
receiving local audio from a second remote user, the second user participating in a second augmented reality view of the live event through an HMD of the second remote user, the second augmented reality view generated from a second physical POV of a second physical location in the real-world venue; modifying the local audio from the second remote user to reflect acoustics of the real-world venue for the first physical location and based on the first physical location and the second physical location; and blending the local audio that is acoustically modified into the audio of the live event that is synchronized and modified.

6 . The method of claim 5 , wherein modifying the local audio further comprises:
determining a distance between the first physical location and the second physical location; determining a delay in the delivery of the local audio based on the distance; and applying the delay to the local audio, wherein the local audio is aligned with a representation of the second remote user in the live view presented to the first remote user.

7 . The method of claim 1 , further comprising:
receiving local audio from a second remote user, the second user participating in a second augmented reality view of the live event through an HMD of the second remote user, the second augmented reality view generated from a second physical POV of a second physical location in the real-world venue; pairing the first remote user and the second remote user over a communication channel; determining a distance between the first physical location and the second physical location; determining a delay in the delivery of the local audio based on the distance; and applying the delay to the local audio delivered over the communication channel, wherein the local audio is aligned with a representation of the second remote user in the live view presented to the first remote user.

8 . A non-transitory computer-readable medium storing a computer program for localizing sound when enabling participation in a live event, the computer-readable medium comprising:
program instructions for establishing a multi-player gaming session controlled by a plurality of players through execution of a gaming application at a server, the multi-player gaming session generating an interactive gaming world within which player participation is enabled, the live event being a real-world venue where the plurality of players is present; program instructions for generating at the server a three dimensional (3D) live view of the real-world venue based on one or more captured video streams, the 3D live view generated for a first physical point-of-view (POV) of the live event, wherein the first physical POV is anchored to a first physical location in the real-world venue; program instructions for generating audio of the live event based on one or more captured audio streams of the live event, the audio being broadcasted over a plurality of sound sources arranged throughout the real-world venue; program instructions for modifying the audio to reflect acoustics of the real-world venue for the first physical location; program instructions for synchronizing the audio that is modified to the 3D live view; and program instructions for streaming the live view and audio that is synchronized and modified via a network to a head mounted display (HMD) of a first remote user for display, the first remote user being located outside of the real-world venue, the live view and audio that are streamed presenting a first augmented reality view of the live event to the first remote user through the HMD.

9 . The computer-readable medium of claim 8 , wherein the program instructions for modifying further comprises:
program instructions for performing a calibration process to determine sound characteristics of the real-world venue for the first physical location and for a plurality of locations of the sound sources in the real-world venue; program instructions for generating acoustic modifications based on the sound characteristics; and program instructions for applying the acoustic modifications when modifying the audio.

10 . The computer-readable medium of claim 9 , wherein the performing a calibration process comprises:
program instructions for broadcasting an audio test pattern from the plurality of sound sources in the real-world venue; program instructions for measuring audio signals received from the audio test pattern that is broadcasted at a plurality of test locations in the real-world venue; program instructions for determining calibration measurements of the audio signals for each of the plurality of test locations; for the first physical location corresponding to the

Assignees

• Sony Interactive Entertainment America Llc

Inventors

• Platt Stuart
• Khan Mohammed

Classifications

• A63F13/52

  involving aspects of the displayed game scene · CPC title

• H04S7/307

  Frequency adjustment, e.g. tone control (H04S7/301 takes precedence) · CPC title

• A63F13/25

  Output arrangements for video game devices · CPC title

• A63F13/216

  using geographical information, e.g. location of the game device or player using GPS · CPC title

• H04S7/301

  Automatic calibration of stereophonic sound system, e.g. with test microphone · CPC title