Method of refocusing images captured by a plenoptic camera and audio based refocusing image system

Patent metadata

Field	Value
Publication number	US-10880466-B2
Application number	US-201615764032-A
Country	US
Kind code	B2
Filing date	Sep 28, 2016
Priority date	Sep 29, 2015
Publication date	Dec 29, 2020
Grant date	Dec 29, 2020

Abstract

A method and system is provided for refocusing images captured by a plenoptic camera. In one embodiment the plenoptic camera is in processing with an audio capture device. The method comprises the steps of determining direction of a dominant audio source associated with an image; creating an audio zoom by filtering out all other audio signals except those associated with said dominant audio source; and performing automatic refocusing of said image based on said created audio zoom.

First claim

The invention claimed is:

1. A method, comprising
refocusing images captured by a plenoptic camera in processing communication with an audio capture device by determining direction of an audio source associated with an image in a scene of interest;
creating an audio zoom based on said audio source by performing beamforming to establish an audio source of interest in said scene of interest and selectively filter out all other audio signals except those associated with said audio source of interest; and
rendering a refocused image and associated audio of said image, wherein said rendering rendering is based on said audio source of interest and its correlated image as determined by said created audio zoom.

2. The method of claim 1 , wherein said audio capture device further comprises a microphone array.

3. The method of claim 2 , wherein said plenoptic camera is a video camera and said captured image is a video image.

4. The method of claim 3 , wherein said audio capture device further comprises a plurality of microphone arrays distributed in a plurality of locations.

5. The method of claim 1 , further comprising performing an audio signal analysis wherein said audio signal analysis comprises determining an angle and width of said audio scene of interest.

6. The method of claim 1 , further comprising determining audio source directions by using audio source localization.

7. The method of claim 1 , further comprising user input, wherein a user selects a particular direction in an audio based identified candidate direction.

8. The method of claim 1 , further comprising user input, wherein a user selects a particular width in an audio based identified candidate direction.

9. An audio based refocusing image system, comprising:
a processor configured for refocusing images captured by a plenoptic video camera in processing communication with an audio capture device by determining direction of an audio source associated with an image in a scene of interest;
said processor also refocusing images captured by a plenoptic camera in processing communication with an audio capture device by determining direction of an audio source associated with an image;
said processor creating an audio zoom based on said audio source by performing beamforming to selectively filter out all other audio signals except those associated with said audio source of interest; and
said processor rendering a refocused image and associated audio of said image, wherein said rendering rendering is based on said audio source of interest and its correlated image as determined by said refocused audio zoom.

10. The audio based automatic refocusing image system of claim 9 further comprising display means for displaying images captured by said plenoptic camera and means for said audio zoom to conduct beamforming on audio scene of interest to selectively filter out all other audio signals except those associated with said dominant audio source.

11. The audio based automatic refocusing image system of claim 9 further comprising a processor in communication with said plenoptic camera, said display and said audio capture device.

12. The audio based automatic refocusing image system of claim 11 , wherein said processor is also configured to identify an audio scene of interest based on direction of audio source.

13. The audio based automatic refocusing image system of claim 9 , wherein said plenoptic camera is a video camera and said captured image is a video image.

Assignees

• Interdigital Ce Patent Holdings

Inventors

• Allie Valérie
• Hellier Pierre
• Duong Quang Khanh Ngoc
• Perez Patrick

Classifications

• H04R1/406 Primary

  microphones · CPC title

• H04N23/67 Primary

  Focus control based on electronic image sensor signals · CPC title

• H04N23/63

  by using electronic viewfinders · CPC title

• H04N23/69

  Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming · CPC title

• G01S3/80

  using ultrasonic, sonic or infrasonic waves · CPC title