Personalized HRTFs via optical capture

Patent metadata

Field	Value
Publication number	US-11778403-B2
Application number	US-201917263125-A
Country	US
Kind code	B2
Filing date	Jul 25, 2019
Priority date	Jul 25, 2018
Publication date	Oct 3, 2023
Grant date	Oct 3, 2023

Abstract

An apparatus and method of generating personalized HRTFs. The system is prepared by calculating a model for HRTFs described as the relationship between a finite example set of input data, namely anthropometric measures and demographic information for a set of individuals, and a corresponding set of output data, namely HRTFs numerically simulated using a high-resolution database of 3D scans of the same set of individuals. At the time of use, the system queries the user for their demographic information, and then from a series of images of the user, the system detects and measures various anthropometric characteristics. The system then applies the prepared model to the anthropometric and demographic data as part of generating a personalized HRTF. In this manner, the personalized HRTF can be generated with more convenience than by performing a high-resolution scan or an acoustic measurement of the user, and with less computational complexity than by numerically simulating their HRTF.

First claim

What is claimed is:

1. A method, comprising:
generating an HRTF calculation system taking anthropometric measurements and, optionally, demographic data of a user as input and returning a personalized HRTF for the user as output,
obtaining 3D scans of a plurality of training subjects,
generating HRTFs for the plurality of training subjects by performing acoustic scattering calculations on anatomical characteristics captured in the 3D scans,
collecting anthropometric measurements and, optionally, demographic data of the plurality of training subjects, and
training the HRTF calculation system to transform the anthropometric measurements and, optionally, demographic data of the plurality of training subjects to the HRTFs.

2. The method of claim 1 , wherein performing acoustic scattering calculations on anatomical characteristics captured in a 3D scan comprises performing a numerical simulation of the sound field around a mesh corresponding to the 3D scan.

3. The method of claim 1 , wherein the anthropometric measurements of the plurality of training subjects are determined from the 3D scans.

4. The method of claim 1 , wherein the training is performed using linear regression with Lasso regularization.

5. The method of claim 1 , further comprising:
collecting a plurality of images of a user;
using the plurality of images of the user to determine anthropometric measurements of the user, and
inputting the anthropometric measurements of the user to the HRTF calculation system to obtain a personalized HRTF for the user.

6. The method of claim 5 , wherein the anthropometric measurements of the user are determined using a convolutional neural network.

7. The method of claim 1 , further comprising:
generating an audio output by applying the personalized HRTF to an audio signal.

8. The method of claim 1 , further comprising:
storing, by a server device, the personalized HRTF; and
transmitting, by the server device, the personalized HRTF to a user device,
wherein the user device generates an audio output by applying the personalized HRTF to an audio signal.

9. The method of claim 1 , wherein an audio signal comprises a plurality of audio objects that includes position information, the method further comprising:
generating a binaural audio output by applying the personalized HRTF to the plurality of audio objects.

10. The method of claim 1 , the method further comprising:
executing, by a server device that is configured to generate the personalized HRTF for the user using the HRTF calculation system, a photogrammetry component, a contextual transformation component, a landmark detection component, and an anthropometry component,
wherein the photogrammetry component receives a plurality of structural imagery of the user, and generates a plurality of camera transforms and a structural image set using a structure-from-motion technique,
wherein the contextual transformation component receives the plurality of camera transforms and the structural image set, and generates a transformed plurality of camera transforms by translating and rotating the plurality of camera transforms using the structural image set,
wherein the landmark detection component receives the structural image set and the transformed plurality of camera transforms, and generates a 3D landmark set that corresponds to anthropometric characteristics of the user identified using the structural image set and the transformed plurality of camera transforms,
wherein the anthropometry component receives the 3D landmark set, and generates anthropometric data from the 3D landmark set, wherein the anthropometric data corresponds to a set of distances and angles measured between individual landmarks of the 3D landmark set, and
wherein the server device generates the personalized HRTF for the user by inputting the anthropometric data into the HRTF calculation system.

11. The method of claim 1 , the method further comprising:
executing, by the server device that is configured to generate the personalized HRTF for the user using the HRTF calculation system, a landmark detection component, a 3D projection component, and an angle and distance measurement component,
wherein the landmark detection component receives a cropped image set of anthropometric landmarks of the user, and generates a set of 2D coordinates of the set of anthropometric landmarks of the user from the cropped image set,
wherein the 3D projection component receives the set of 2D coordinates and a plurality of camera transforms, and generates a set of 3D coordinates that correspond to the set of 2D coordinates of each of the anthropometric landmarks in 3D space using the camera transforms,
wherein the angle and distance measurement component receives the set of 3D coordinates, and generates anthropometric data from the set of 3D coordinates, wherein the anthropometric data correspond to angles and distances of the anthropometric landmarks in the set of 3D coordinates,
wherein the server device generates the personalized HRTF for the user by inputting the anthropometric data into the HRTF calculation system.

12. A non-transitory computer readable medium storing a computer program that, when executed by a processor, controls an apparatus to execute processing including the method of claim 1 .

13. An apparatus for generating head-related transfer functions (HRTFs), the apparatus comprising:
at least one processor; and
at least one memory,
wherein the at least one processor is configured to control the apparatus to:
generate an HRTF calculation system taking anthropometric measurements and, optionally, demographic data of a user as input and returning a personalized HRTF for the user as output,
obtain 3D scans of a plurality of training subjects,
generate HRTFs for the plurality of training subjects by performing acoustic scattering calculations on anatomical characteristics captured in the 3D scans,
collect anthropometric measurements and, optionally, demographic data of the plurality of training subjects, and
train the HRTF calculation system to transform the anthropometric measurements and, optionally, demographic data of the plurality of training subjects to the HRTFs.

14. The apparatus of claim 13 , wherein the at least one processor is further configured to control the apparatus to perform acoustic scattering calculations on anatomical characteristics captured in a 3D scan by performing a numerical simulation of the sound field around a mesh corresponding to the 3D scan.

15. The apparatus of claim 13 , wherein the at least one processor is further configured to control the apparatus to determine the anthropometric measurements of the plurality of training subjects from the 3D scans.

16. The apparatus of claim 13 , the apparatus further comprising:
a user input device that is configured to collect a plurality of images of a user, wherein the at least one processor is further configured to use a plurality of images of a user captured by the user input device to determine anthropometric measurements of the user, and to input the anthropometric measurements of the user to the HRTF calculation system to obtain a personalized HRTF for the user.

17. The apparatus of claim 13 , further comprising:
a user output device that is configured to generate an audio output by applying the personalized HRTF to an audio signal.

18. The apparatus of claim 13 , further comprising:
a server device that is configured to generate the HRTF calculation system, to generate the personalized HRTF, to store the personalized HRTF, and to

Assignees

• Dolby Laboratories Licensing Corp

Inventors

• Joyner Mcgregor Steele
• Brandmeyer Alex
• Daly Scott
• Baker Jeffrey Ross
• Fanelli Andrea
• Crum Poppy Anne Carrie

Classifications

• H04S7/301 Primary

  Automatic calibration of stereophonic sound system, e.g. with test microphone · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title