Augmented reality audio mixing

Patent metadata

Field	Value
Publication number	US-2020042284-A1
Application number	US-201916601702-A
Country	US
Kind code	A1
Filing date	Oct 15, 2019
Priority date	Apr 2, 2018
Publication date	Feb 6, 2020
Grant date	—

Abstract

Augmented reality enables an operator to visualize values of parameters of audio channels during audio mixing. A heads-up display worn by the operator, such as smart glasses, displays virtual graphical objects representing the parameters so that they appear within a three-dimensional space surrounding the operator and an audio mixing console. Parameter values are represented by the location, size, and other attributes of the virtual graphical objects. The operator adjusts the parameter values using physical and touch controls on the console and by manipulating the virtual objects with a body part such as a finger. Sensors mounted on the heads-up display and on other body parts capture position and movement of the operator in real time and send the captured data to a system running augmented reality control software. Graphical user interfaces of a media processing application controlled by the console and of plug-in modules may also be displayed on the heads-up display.

First claim

What is claimed is:

1 . A method of mixing a plurality of audio channels of a media project, the method comprising:
providing a physical mixing console for mixing the plurality of audio channels of the media project; providing smart glasses for an operator of the physical mixing console, wherein the physical mixing console and the smart glasses are in data communication with a computer hosting augmented reality software; providing to the augmented reality software a mapping of a physical space surrounding the operator and the physical mixing console, wherein the mapping includes a mapping of a physical location of the physical mixing console within the physical space; and while the operator is wearing the smart glasses, displaying on the smart glasses a graphical representation in virtual space of a first element of a user interface for mixing the plurality of audio channels, wherein a location within the virtual space at which the first element of the user interface is displayed is chosen to avoid covering an object which needs to be visible to the operator in at least one of the physical space and the virtual space.

2 . The method of claim 1 , wherein the object which needs to be visible to the operator is the physical mixing console.

3 . The method of claim 1 , further comprising displaying on the smart glasses a graphical representation in virtual space of a second element of the user interface for mixing the plurality of audio channels, wherein the object which needs to be visible to the operator is the first element of the user interface for mixing the plurality of audio channels.

4 . The method of claim 1 , wherein the first user interface element comprises a virtual monitor displaying a graphical user interface of a digital audio workstation that is being used to mix the plurality of audio channels.

5 . The method of claim 4 , wherein the virtual monitor appears to be displayed on a wall of the physical space.

6 . The method of claim 1 , wherein the physical space includes a screen for displaying a synchronized video component of the media project while the operator is mixing the plurality of audio channels of the media project.

7 . The method of claim 1 , wherein the first user interface element enables the operator to interact with one or more parameters of the plurality of audio channels.

8 . The method of claim 1 , further comprising displaying on the smart glasses a user interface element in the virtual space enabling the operator to control equalization parameters of the plurality of audio channels.

9 . The method of claim 1 , further comprising displaying on the smart glasses a user interface element in virtual space enabling the operator to control a dynamics parameter of the one or more audio channels.

10 . The method of claim 1 , further comprising displaying on the smart glasses a user interface element in virtual space showing three-dimensional spectrograms of one or more of the one or more audio channels.

11 . The method of claim 1 , further comprising displaying on the smart glasses a user interface element in virtual space showing filter response curves of one or more of the one or more audio channels.

12 . A system for audio mixing comprising:
a control system in data communication with augmented reality smart glasses and with a physical mixing console, wherein the control system includes augmented reality software, which when executed on the control system, cause the control system to enable a method of mixing a plurality of audio channels of a media project, the method comprising:
the control system receiving a mapping of a physical space surrounding an operator and the physical mixing console, wherein the mapping includes a mapping of a physical location of the physical mixing console within the physical space; and
while the operator is wearing the smart glasses, displaying on the smart glasses a graphical representation in virtual space of a first element of a user interface for mixing the plurality of audio channels, wherein a location within the virtual space at which the first element of the user interface is displayed is chosen to avoid covering an object which needs to be visible to the operator in at least one of the physical space and the virtual space.

13 . The system of claim 12 , wherein the object which needs to be visible to the operator is the physical mixing console.

14 . The system of claim 12 , further comprising displaying on the smart glasses a graphical representation in virtual space of a second element of the user interface for mixing the plurality of audio channels, wherein the object which needs to be visible to the operator is the first element of the user interface for mixing the plurality of audio channels.

15 . The system of claim 12 , wherein the first user interface element comprises a virtual monitor displaying a graphical user interface of a digital audio workstation that is being used to mix the plurality of audio channels.

16 . The system of claim 15 , wherein the virtual monitor appears to be displayed on a wall of the physical space.

17 . The system of claim 12 , wherein the physical space includes a screen for displaying a synchronized video component of the media project while the operator is mixing the plurality of audio channels of the media project.

18 . The system of claim 12 , wherein the first user interface element enables the operator to interact with one or more parameters of the plurality of audio channels.

19 . The system of claim 12 , further comprising displaying on the smart glasses a user interface element in the virtual space enabling the operator to control equalization parameters of the plurality of audio channels.

20 . The system of claim 12 , further comprising displaying on the smart glasses a user interface element in virtual space enabling the operator to control a dynamics parameter of the one or more audio channels.

Assignees

• Avid Tech Inc

Inventors

• Milne Steven H
• Wilson Stephen
• Jones Edward
• Boyer Robert
• Magleby David K
• Trama Eduardo Dias
• Baranov Konstantine

Classifications

• G06T15/20

  Perspective computation · CPC title

• H04S2400/01

  Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved · CPC title

• H04S3/008

  in which the audio signals are in digital form, i.e. employing more than two discrete digital channels (data reduction aspects thereof based on psychoacoustics G10L19/02) · CPC title

• G06F3/014

  Hand-worn input/output arrangements, e.g. data gloves · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title