Audio alteration techniques

Patent metadata

Field	Value
Publication number	US-9930310-B2
Application number	US-55638009-A
Country	US
Kind code	B2
Filing date	Sep 9, 2009
Priority date	Sep 9, 2009
Publication date	Mar 27, 2018
Grant date	Mar 27, 2018

Abstract

A method of altering audio output from an electronic device based on image data is provided. In one embodiment, the method includes acquiring image data and determining one or more characteristics of the image data. Such characteristics may include sharpness, brightness, motion, magnification, zoom setting, and so forth, as well as variation in any of the preceding characteristics. The method may also include producing audio output, wherein at least one characteristic of the audio output is determined based on one or more of the image data characteristics. Various audio output characteristics that may be varied based on the video data characteristics may include, for instance, pitch, reverberation, tempo, volume, filter frequency response, added sound effects, or the like. Additional methods, devices, and manufactures are also disclosed.

First claim

What is claimed is:

1. A method comprising:
receiving, at an electronic device, a plurality of video data frames representing an object;
receiving, at the electronic device, an audio signal associated with the plurality of video data frames, wherein the audio signal includes a first audio property and a second audio property;
processing, by the electronic device, the video data frames, wherein processing the video data frames includes encoding the video data frames via a video encoder of the electronic device to provide encoding data, wherein the encoding data includes a motion vector calculated by the video encoder for the object, and wherein the motion vector includes a horizontal directionality and a vertical directionality; and
altering, by the electronic device, the audio signal associated with the plurality of video data frames, wherein altering the audio signal includes:
adjusting the first audio property based on the horizontal directionality of the motion vector, and
adjusting the second audio property based on the vertical directionality of the motion vector.

2. The method of claim 1 , wherein the audio signal further includes a third audio property that varies based on a total amount of motion in the received plurality of video data frames.

3. The method of claim 1 , wherein adjusting the first audio property or the second audio property includes varying at least one of a volume, a pitch, or a tempo of the audio signal.

4. The method of claim 1 , further comprising outputting the audio signal from the electronic device, wherein outputting the audio signal includes at least one of playing the audio signal to a user via a speaker or outputting electronic data encoding the audio signal to an additional electronic device.

5. The method of claim 1 , further comprising outputting video generated from the plurality of video data frames to a user via a display of the electronic device.

6. The method of claim 1 , wherein receiving the plurality of video data frames includes acquiring the plurality of video data frames via an image capture device of the electronic device.

7. An electronic device comprising:
an image capture device disposed in a housing;
an image signal processor disposed in the housing;
a video encoder disposed in the housing; and
audio processing logic disposed in the housing;
wherein the electronic device is configured to acquire a plurality of video data frames representing an object via the image capture device, to acquire an audio signal associated with the plurality of video data frames and having a first audio property and a second audio property, to encode the video data frames such that the encoding data includes a motion vector having a horizontal directionality and a vertical directionality calculated by the video encoder for motion of the object, to adjust the first audio property via the audio processing logic based on the horizontal directionality of the motion vector, and to adjust the second audio property via the audio processing logic based on the vertical directionality of the motion vector.

8. The electronic device of claim 7 , wherein the electronic device is configured to determine at least one of sharpness, brightness, or color statistics for the plurality of video data frames, and to adjust at least one of the first audio property or the second audio property via the audio processing logic based on the at least one of sharpness, brightness, or color statistics for the plurality of video data frames.

9. The electronic device of claim 8 , wherein adjusting at least one of the first audio property or the second audio property includes at least one of variation of a pitch of the audio signal based on the directionality of the motion vector or variation of a volume of the audio signal based on the directionality of the motion vector.

10. The electronic device of claim 8 , wherein adjusting the first audio property or the second audio property includes changing a tempo of music played from the electronic device, and wherein changing the tempo is based on the motion of the object.

11. The electronic device of claim 7 , wherein the electronic device is configured to output the adjusted audio signal to a user during encoding of the acquired video data frames.

12. The electronic device of claim 7 , wherein the video encoder includes an H.264 video encoder.

13. A method comprising:
receiving, at an electronic device, image data representing an object;
receiving, at the electronic device, an audio signal associated with the image data, wherein the audio signal includes a first audio property and a second audio property;
analyzing the image data via image processing logic of the electronic device; and
altering a characteristic of the audio signal based on the analyzed image data;
wherein analyzing the image data includes determining motion of the object represented in the image data and calculating a motion vector having a horizontal directionality and a vertical directionality for the object, and wherein altering the characteristic of the audio signal includes altering the first audio property based on the vertical directionality of the motion vector or altering the second audio property based on the horizontal directionality of the motion vector.

14. The method of claim 13 , wherein receiving the image data at the electronic device includes receiving a plurality of static images.

15. The method of claim 14 , comprising sequentially displaying the plurality of static images and outputting audio to a user therewith, wherein the output audio varies based on the analysis of the plurality of static images via the image processing logic.

16. The method of claim 13 , wherein analyzing the image data includes determining at least one of sharpness, brightness, or color statistics, and wherein altering the characteristic of the audio signal includes altering at least one of the first audio property or the second audio property of the audio signal based on the at least one of sharpness, brightness, or color statistics.

17. The method of claim 16 , wherein determining at least one of sharpness, brightness, or color statistics includes determining at least one of sharpness, brightness, or color statistics for one or more of a plurality of still images or a sequence of frames in video data.

18. A manufacture comprising:
one or more non-transitory, computer-readable storage media having application instructions encoded thereon for execution by a processing system, the application instructions comprising:
instructions for acquiring image data representing an object;
instructions for acquiring an audio signal associated with the image data, wherein the audio signal includes a first audio property, a second audio property, and a third audio property;
instructions for receiving an image attribute from image processing logic configured to encode the acquired image data;
instructions for altering the audio signal based on a motion vector calculated for motion of the object represented in the image data, wherein the motion vector includes a horizontal directionality and a vertical directionality, and wherein altering the audio signal includes adjusting the first audio property based on the horizontal directionality of the motion vector and adjusting the second audio property based on the vertical directionality of the motion vector; and
instructions for generating an additional audio effect that alters the audio signal, wherein altering the audio signal includes adjusting the third audio property in response to a physical motion of a

Assignees

• Lindahl Aram
• Chiu Kelvin
• Apple Inc

Inventors

• Lindahl Aram
• Chiu Kelvin

Classifications

• H04N9/8063 Primary

  using time division multiplex of the PCM audio and PCM video signals · CPC title

• H04N21/440236

  by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text · CPC title

• H04N21/47

  End-user applications · CPC title

• H04N21/4524

  involving the geographical location of the client (retrieval from the Internet by querying based on geographical locations G06F16/9537; systems specially adapted for using geographical information in broadcast systems H04H60/70; protocols in which the network application is adapted for the location of the user terminal in communication control or processing H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00) · CPC title

• H04N21/439

  Processing of audio elementary streams · CPC title