Augmented reality audio mixing

Patent metadata

Field	Value
Publication number	US-10466960-B2
Application number	US-201815943153-A
Country	US
Kind code	B2
Filing date	Apr 2, 2018
Priority date	Apr 2, 2018
Publication date	Nov 5, 2019
Grant date	Nov 5, 2019

Abstract

Augmented reality enables an operator to visualize values of parameters of audio channels during audio mixing. A heads-up display worn by the operator, such as smart glasses, displays virtual graphical objects representing the parameters so that they appear within a three-dimensional space surrounding the operator and an audio mixing console. Parameter values are represented by the location, size, and other attributes of the virtual graphical objects. The operator adjusts the parameter values using physical and touch controls on the console and by manipulating the virtual objects with a body part such as a finger. Sensors mounted on the heads-up display and on other body parts capture position and movement of the operator in real time and send the captured data to a system running augmented reality control software. Graphical user interfaces of a media processing application controlled by the console and of plug-in modules may also be displayed on the heads-up display.

First claim

What is claimed is:

1. A method of mixing a plurality of audio channels of a media project, the method comprising:
providing a physical mixing console for mixing the plurality of audio channels of the media project;
providing smart glasses for an operator of the physical mixing console, wherein the physical mixing console and the smart glasses are in data communication with a computer hosting augmented reality software;
while the operator is wearing the smart glasses, displaying on the smart glasses a graphical representation of a spatial location of a source of a given audio channel, wherein the graphical representation of the spatial location of the source of the given audio channel appears to the operator to be positioned at a spatial location within a three-dimensional space surrounding the operator and the physical mixing console; and
enabling the operator to adjust the spatial location of the source of the given audio channel with respect to the three-dimensional space, wherein the graphical representation of the spatial location of the source of the given audio channel is updated in response to an operator using a combination of a knob and a joystick of the physical mixing console to adjust the spatial location.

2. The method of claim 1 , wherein the operator is able to adjust the location of the source of the given audio channel by manipulating a physical control on the physical mixing console.

3. The method of claim 1 , wherein the operator is able to adjust the spatial location of the source of the given audio channel by touching a touchscreen control on the physical mixing console.

4. The method of claim 1 , wherein the operator is able to adjust the spatial location of the source of the given audio channel by using gestures that appear to interact in the three-dimensional space with the graphical representation of the spatial location of the source of the given audio channel.

5. The method of claim 1 wherein a size of the graphical representation of the spatial location of the source of the given audio channel is indicative of a distance of the source of the given audio channel from the operator.

6. The method of claim 1 wherein a shape of the graphical representation is indicative of a value of a parameter of the given audio channel.

7. The method of claim 1 wherein a color of a graphical representation is indicative of a value of a parameter of the given audio channel.

8. The method of claim 1 , wherein the graphical representation of the spatial location of the given audio channel includes a name of the given audio channel.

9. The method of claim 1 wherein:
the media project comprises time-synchronous video and audio;
the time-synchronous video is displayed on a display within the three-dimensional space surrounding the operator and the mixing console;
a source object for the given audio channel is depicted in the displayed time-synchronous video; and
the spatial location of the graphical representation of the source of the given audio channel appears to coincide with a spatial location within the displayed time-synchronous video of the depicted source object.

10. The method of claim 1 , wherein the graphical representation of the spatial location of the source of the given audio channel is displayed within a graphical user interface of a media processing application, and the graphical user interface of the media processing application appears to the operator to be positioned on a surface of the three-dimensional space surrounding the operator.

11. The method of claim 1 , further comprising displaying on the smart glasses graphical representations of values of a plurality of audio mixing.

12. The method of claim 1 , wherein the computer is embedded within the physical mixing console.

13. A system for audio mixing comprising:
a control system in data communication with augmented reality smart glasses and with a physical mixing console, wherein:
the augmented reality smart glasses include a three-dimensional position sensor;
the physical mixing console is configured to enable an operator using a combination of a knob and a joystick to adjust a spatial location of a source of a given audio channel of media project;
the control system is configured to:
receive from the physical mixing console information defining an adjusted spatial position of the source of the given audio channel;
in response to receiving the information defining the adjusted spatial position of the source of the given audio channel, generate data representing a graphical representation of the adjusted spatial position of the source of the given audio channel; and
send the data representing the graphical representation of the adjusted spatial position of the given audio channel to the augmented reality smart glasses; and

the augmented reality smart glasses receive the data representing the graphical representation of the adjusted spatial position of the source of the given audio channel and display the graphical representation of the adjusted spatial position of the source of the given audio channel so that it appears to the operator to be located within a three-dimensional space that surrounds the operator and the mixing console.

14. The system of claim 13 , wherein the operator uses a control of the physical mixing console to adjust the spatial location of the given audio channel.

15. The system of claim 13 , further comprising a three-dimensional position sensor in data communication with the control system, wherein:
the three-dimensional position sensor tracks a movement of the operator and sends data representing the tracked movement to the control system;
the control system in real-time:
interprets the tracked movement as an instruction to adjust the spatial position of the source of the given audio channel and generates in real-time data representing a graphical representation corresponding to an adjusted spatial position of the source of the given audio channel; and
sends the data representing the graphical representation of the adjusted spatial position to the augmented reality smart glasses; and

the augmented reality smart glasses receive the data representing the graphical representation of the adjusted spatial position and display the graphical representation of the adjusted spatial position.

Assignees

• Avid Tech Inc

Inventors

• Milne Steven H
• Wilson Stephen
• Jones Edward
• Boyer Robert
• Magleby David K
• Trama Eduardo Dias
• Baranov Konstantine

Classifications

• G06T15/20

  Perspective computation · CPC title

• H04S3/008

  in which the audio signals are in digital form, i.e. employing more than two discrete digital channels (data reduction aspects thereof based on psychoacoustics G10L19/02) · CPC title

• H04S2400/01

  Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved · CPC title

• G06F3/014

  Hand-worn input/output arrangements, e.g. data gloves · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title