Information processing device, information processing method, and information processing program

Patent metadata

Field	Value
Publication number	US-11595772-B2
Application number	US-201917282705-A
Country	US
Kind code	B2
Filing date	Oct 3, 2019
Priority date	Oct 10, 2018
Publication date	Feb 28, 2023
Grant date	Feb 28, 2023

Abstract

An information processing device (100) according to the present disclosure includes: an acquisition unit (141) configured to acquire a first image including a content image of an ear of a user; and a calculation unit (142) configured to calculate, based on the first image acquired by the acquisition unit (141), a head-related transfer function corresponding to the user by using a learned model having learned to output a head-related transfer function corresponding to an ear when an image including a content image of the ear is input.

First claim

The invention claimed is:

1. An information processing device comprising:
circuitry configured to function as:
an acquisition unit configured to acquire a first image including a content image of an ear of a user;
a calculation unit configured to calculate, based on the first image acquired by the acquisition unit, a head-related transfer function corresponding to the user by using a learned model having learned to output a head-related transfer function corresponding to an ear when an image including a content image of the ear is input; and
a first learning unit configured to generate an ear parameter estimation model by learning a relation between a plurality of ear images obtained by changing:
texture of three-dimensional data of the ear or a head, or luminance in rendering, and
an ear parameter common to the ear images,
wherein the texture includes a skin color of the ear or of the head.

2. The information processing device according to claim 1 , wherein
the acquisition unit acquires an ear parameter that is a variable representing a characteristic of the ear included in the first image, and
the calculation unit calculates the head-related transfer function corresponding to the user by inputting the ear parameter to the learned model.

3. The information processing device according to claim 2 , wherein the acquisition unit acquires the ear parameter of the ear included in the first image by using an ear parameter estimation model having learned to output an ear parameter corresponding to an ear when an image including a content image of the ear is input.

4. The information processing device according to claim 3 , wherein the first learning unit is configured to generate the ear parameter estimation model by learning a relation between an image including a content image of an ear and an ear parameter of the ear.

5. The information processing device according to claim 4 , wherein the first learning unit generates the ear parameter estimation model by learning a relation between the ear parameter and an ear image obtained by rendering three-dimensional data of the ear generated based on the ear parameter.

6. The information processing device according to claim 5 , wherein the first learning unit generates the ear parameter estimation model by learning the relation between the plurality of ear images obtained by changing a camera angle in rendering.

7. The information processing device according to claim 4 , further comprising a second learning unit configured to generate the learned model by learning a relation between an image including a content image of an ear and a head-related transfer function corresponding to the ear.

8. The information processing device according to claim 7 , wherein the second learning unit performs acoustic simulation for three-dimensional data obtained by synthesizing three-dimensional data of the ear generated based on the ear parameter and three-dimensional data of a head, and generates the learned model by learning a relation between a head-related transfer function obtained through the acoustic simulation and the ear parameter.

9. The information processing device according to claim 8 , wherein the second learning unit compresses an information amount of the head-related transfer function obtained through the acoustic simulation, and generates the learned model by learning a relation between the compressed head-related transfer function and the ear parameter.

10. The information processing device according to claim 8 , wherein the second learning unit sets a hearing point of three-dimensional data of the ear generated based on the ear parameter, and performs the acoustic simulation by using the set hearing point.

11. The information processing device according to claim 1 , further comprising a preprocessing unit configured to specify a content image of an ear of the user in a second image including a content image of the entire head of the user, and detect a specified range as the first image, wherein the acquisition unit acquires the first image detected by the preprocessing unit.

12. The information processing device according to claim 11 , wherein the preprocessing unit specifies the range based on a relation between a feature point of the head of the user included in the second image and a posture of the user.

13. The information processing device according to claim 12 , wherein when the range cannot be specified based on the relation between the feature point of the head of the user included in the second image and the posture of the user, the preprocessing unit newly requests acquisition of an image different from the second image and including a content image of the entire head of the user.

14. The information processing device according to claim 11 , wherein the preprocessing unit specifies a content image of an ear of the user by correcting rotation of the second image based on correction information included in the second image, and detects a specified range as the first image.

15. An information processing method by which a computer performs:
acquiring a first image including a content image of an ear of a user;
calculating, based on the acquired first image, a head-related transfer function corresponding to the user by using a learned model having learned to output a head-related transfer function corresponding to an ear when an image including a content image of the ear is input; and
generating an ear parameter estimation model by learning a relation between a plurality of ear images obtained by changing:
texture of three-dimensional data of the ear or a head, or luminance in rendering, and
an ear parameter common to the ear images,
wherein the texture includes a skin color of the ear or of the head.

16. A non-transitory computer-readable storage medium encoded with executable instructions that, when executed by at least one processor, cause the at least one processor to perform:
acquiring a first image including a content image of an ear of a user;
calculating, based on the first image acquired, a head-related transfer function corresponding to the user by using a learned model having learned to output a head-related transfer function corresponding to an ear when an image including a content image of the ear is input; and
generating an ear parameter estimation model by learning a relation between a plurality of ear images obtained by changing:
texture of three-dimensional data of the ear or a head, or luminance in rendering, and
an ear parameter common to the ear images,
wherein the texture includes a skin color of the ear or of the head.

Assignees

• Sony Group Corp

Inventors

• Fukuda Kazumi
• Magariyachi Tetsu

Classifications

• G06V20/647

  by matching two-dimensional images to three-dimensional objects · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• H04R5/033

  Headphones for stereophonic communication {(details thereof, e.g. relating to batteries, cables or control elements H04R1/10)} · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• H04S2420/01

  Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD] · CPC title