Audio event detection for automatic plenoptic video refocusing

Patent metadata

Field	Value
Publication number	US-10674057-B2
Application number	US-201615278664-A
Country	US
Kind code	B2
Filing date	Sep 28, 2016
Priority date	Sep 29, 2015
Publication date	Jun 2, 2020
Grant date	Jun 2, 2020

Abstract

A plenoptic camera and associated method is provided. The camera has an array of sensors for generating digital images. The images have associated audio signals. The array of sensors are configured to capture digital images associated with a default spatial coordinate and are also configured to receive control input from a processor to change focus from said default spatial coordinate to a new spatial coordinate based on occurrence of an event at said new spatial coordinate.

First claim

What is claimed is:

1. A plenoptic camera having an array of sensors for generating digital images, said images having associated audio signals, comprising:
said array of sensors configured to capture digital images associated with a default spatial coordinate;
said array of sensors configured to receive control input from a processor to change focus from said default spatial coordinate to a new spatial coordinate based on occurrence of a classifier including a preselected or priority detected event, at said new spatial coordinate, said classifier being selected among a plurality of classifiers occurring concurrently based on a selected order of classifiers on a priority list.

2. The system of claim 1 , wherein at least one sensor is in communication with a processor.

3. The system of claim 2 , wherein said event is associated with occurrence of at least one classifier as detected by said sensor.

4. The system of claim 3 , wherein said audio signals are captured by a plurality of microphones.

5. The system of claim 2 , wherein more than one classifier exists and the order of refocusing is based on a selected order of classifiers on a priority list.

6. The system of claim 5 , wherein at least one classifier is an audio event.

7. The system of claim 6 , wherein said audio event includes one of: laughs, claps, or sound of a ball.

8. The system of claim 6 , wherein said audio event is any abnormally loud sound above a preselected decibel threshold.

9. The system of claim 1 , wherein said event is associated with occurrence of at least one classifier as detected by said sensor.

10. The system of claim 1 , wherein said audio signals are captured by a plurality of microphones.

11. A method of capturing an image using a plenoptic camera focused on a default spatial coordinate, comprising:
receiving via a processor a plurality of digital images from said plenoptic camera;
receiving via said processor a plurality of audio signals associated with said digital images;
determining by said processor occurrence of a classifier including a preselected or priority detected event and changing focus from said default spatial coordinate to a new spatial coordinate associated with said occurrence of said event classifier, said classifier being selected among a plurality of classifiers occurring concurrently based on a selected order of classifiers on a priority list.

12. The method of claim 11 , wherein said default spatial coordinate is associated with a venue and said audio signals are received from a plurality of microphones disposed in different locations in said venue further comprising:
receiving by said processor a plurality of calibration scores from said plurality of microphones; determining by said processor an interest score associated with location of each of said microphones; calculating by said processor a threshold audio range for each of said microphones based on said calibration score and said interest score; said processor changing focus and delivery of digital images and associated audio signal from said default spatial coordinate to a new spatial coordinate when an audio range threshold has been exceeded.

13. The method of claim 12 , wherein said new spatial coordinate is being associated with location of said microphone associated with said exceeded audio range threshold.

14. The method of claim 13 , wherein if more than one threshold range is exceeded, said processor provide said new spatial coordinate based on the microphone associated with the highest interest score.

15. The method of claim 14 , wherein said processor categorizes said microphones by interest scores.

16. The method of claim 15 , wherein said processor category is ranged from highly rated to lowly rated.

17. The method of claim 16 , wherein a refocusing plane is calculated by said processor for delivery of images when threshold audio range has been exceeded by more than one microphone.

18. The method of claim 17 , wherein an optimal position of said refocusing plane is computed by said processor from a subset of microphone position with relatively high interest score.

19. The method of claim 18 , wherein said refocusing plane can be optimized by said processor by computing a weighted regression value.

20. The method of claim 19 , wherein said weighted regression value is estimated based on the value of minimized mean square roots of each microphone and their associated interest score.

Assignees

• Interdigital Ce Patent Holdings

Inventors

• Hellier Pierre
• Duong Quang Khanh Ngoc
• Allie Valerie
• Leyendecker Philippe

Classifications

• H04S2400/15

  Aspects of sound capture and related signal processing for recording or reproduction · CPC title

• H04R2499/11

  Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's · CPC title

• H04R2227/003

  Digital PA systems using, e.g. LAN or internet · CPC title

• H04R1/406

  microphones · CPC title

• H04N5/232 Primary

  Electricity · mapped topic