Sound localization in an augmented reality view of a live event held in a real-world venue

Patent metadata

Field	Value
Publication number	US-11179635-B2
Application number	US-201715730403-A
Country	US
Kind code	B2
Filing date	Oct 11, 2017
Priority date	Oct 11, 2017
Publication date	Nov 23, 2021
Grant date	Nov 23, 2021

Abstract

A method for localizing sound when enabling participation in a live event, including establishing a multi-player gaming session of a gaming application that generates an interactive gaming world, the live event being a real-world venue where players playing the gaming application are present. A live view of the venue is generated based on captured video streams, and generated for a physical POV anchored to a physical location in the venue. Audio is generated for the live event, and modified to reflect acoustics of the real-world venue for the physical location. The modified audio is synchronized to the live view. The live view and synchronized audio are streamed to an HMD of a remote user located outside the venue, and presents an augmented reality view of the live event.

First claim

What is claimed is:

1. A method for localizing sound when enabling participation in a live event, comprising:
establishing a multi-player gaming session controlled by a plurality of players through execution of a gaming application at a server, the multi-player gaming session generating an interactive gaming world within which player participation is enabled, the live event being a real-world venue where the plurality of players is present;
generating at the server a three dimensional (3D) live view of the real-world venue based on one or more captured video streams, the 3D live view generated for a first physical point-of-view (POV) of the live event, wherein the first physical POV is anchored to a first physical location in the real-world venue other than where a captured video stream is captured, wherein the first physical location is a seat in the real-world venue where a spectator may be physically present;
generating audio of the live event based on one or more captured audio streams of the live event, the audio being broadcasted over a plurality of sound sources arranged throughout the real-world venue;
modifying the audio to reflect acoustics of the real-world venue for the first physical location;
synchronizing the audio that is modified to the 3D live view;
streaming the 3D live view and the audio that is synchronized and modified via a network to a head mounted display (HMD) of a first remote user for display, the first remote user being located outside of the real-world venue, the 3D live view and the audio that is synchronized and modified that are streamed presenting a first augmented reality view of the live event to the first remote user through the HMD:
capturing audio of the first remote user; and
broadcasting the audio of the first remote user from a speaker located at the seat of the first physical location.

2. The method of claim 1 , wherein the modifying the audio includes:
performing a calibration process to determine sound characteristics of the real-world venue for the first physical location and for a plurality of locations of the plurality of sound sources in the real-world venue;
generating acoustic modifications based on the sound characteristics; and
applying the acoustic modifications when performing the modifying the audio.

3. The method of claim 2 , wherein the performing the calibration process includes:
broadcasting an audio test pattern from the plurality of sound sources in the real-world venue;
measuring audio signals received from the audio test pattern that is broadcasted at a plurality of test locations in the real-world venue;
determining calibration measurements of the audio signals that are measured for each of the plurality of test locations;
for the first physical location corresponding to the first physical POV in the real world venue, determining localized calibration measurements for the first physical location through interpolation of first calibration measurements from a first set of test locations relevant to the first physical location; and
generating the acoustic modifications for the first physical location based on the localized calibration measurements.

4. The method of claim 2 , wherein the sound characteristics include timing characteristics, reflection characteristics, refraction characteristics, interference characteristics, and attenuation characteristics.

5. The method of claim 1 , further comprising:
receiving local audio from a second remote user, the second remote user participating in a second augmented reality view of the live event through an HMD of the second remote user, the second augmented reality view generated from a second physical POV of a second physical location in the real-world venue;
modifying the local audio from the second remote user to reflect the acoustics of the real-world venue for the first physical location and based on the first physical location and the second physical location; and
blending the local audio that is acoustically modified into the audio of the live event that is synchronized and modified.

6. The method of claim 5 , wherein modifying the local audio further comprises:
determining a distance between the first physical location and the second physical location;
determining a delay in the delivery of the local audio based on the distance; and
applying the delay to the local audio, wherein the local audio is aligned with a representation of the second remote user in the 3D live view presented to the first remote user.

7. The method of claim 1 , further comprising:
receiving local audio from a second remote user, the second user participating in a second augmented reality view of the live event through an HMD of the second remote user, the second augmented reality view generated from a second physical POV of a second physical location in the real-world venue;
pairing the first remote user and the second remote user over a communication channel;
determining a distance between the first physical location and the second physical location;
determining a delay in the delivery of the local audio based on the distance; and
applying the delay to the local audio delivered over the communication channel, wherein the local audio is aligned with a representation of the second remote user in the 3D live view presented to the first remote user.

8. A non-transitory computer-readable medium storing a computer program for localizing sound when enabling participation in a live event, the non-transitory computer-readable medium comprising:
program instructions for establishing a multi-player gaming session controlled by a plurality of players through execution of a gaming application at a server, the multi-player gaming session generating an interactive gaming world within which player participation is enabled, the live event being a real-world venue where the plurality of players is present;
program instructions for generating at the server a three dimensional (3D) live view of the real-world venue based on one or more captured video streams, the 3D live view generated for a first physical point-of-view (POV) of the live event, wherein the first physical POV is anchored to a first physical location in the real-world venue other than where a captured video stream is captured, wherein the first physical location is a seat in the real-world venue where a spectator may be physically present;
program instructions for generating audio of the live event based on one or more captured audio streams of the live event, the audio being broadcasted over a plurality of sound sources arranged throughout the real-world venue;
program instructions for modifying the audio to reflect acoustics of the real-world venue for the first physical location;
program instructions for synchronizing the audio that is modified to the 3D live view;
program instructions for streaming the 3D live view and the audio that is synchronized and modified via a network to a head mounted display (HMD) of a first remote user for display, the first remote user being located outside of the real-world venue, the 3D live view and the audio that is synchronized and modified that are streamed presenting a first augmented reality view of the live event to the first remote user through the HMD:
program instructions for capturing audio of the first remote user; and
program instructions for broadcasting the audio of the first remote user from a speaker located at the seat of the first physical location.

9. The non-transitory computer-readable medium of claim 8 , wherein the program instructions for modifying the audio includes:
program instructions for performing a calibration process to determine sound characteristics of the real-world venue for the first physical

Assignees

• Sony Interactive Entertainment LLC

Inventors

• Platt Stuart
• Khan Mohammed

Classifications

• H04S7/304 Primary

  For headphones · CPC title

• H04S7/306

  For headphones · CPC title

• H04R27/00

  Public address systems (circuits for preventing acoustic reaction H04R3/02; circuits for distributing signals to loudspeakers H04R3/12; {monitoring or testing arrangements for public address systems H04R29/007}; amplifiers H03F) · CPC title

• H04R2227/003

  Digital PA systems using, e.g. LAN or internet · CPC title

• H04S2400/15

  Aspects of sound capture and related signal processing for recording or reproduction · CPC title