Devices with enhanced audio

Patent metadata

Field	Value
Publication number	US-11640275-B2
Application number	US-202117369374-A
Country	US
Kind code	B2
Filing date	Jul 7, 2021
Priority date	Jul 28, 2011
Publication date	May 2, 2023
Grant date	May 2, 2023

Abstract

A system for enhancing audio including a plurality of sensors, an output device, and a processor in communication with the plurality of sensors and the output device. The processor is configured to process data captured by the plurality of sensors, and based on that, modify an output of the output device. The processor also is configured to determine whether there are a plurality of users associated with a video conferencing session, determine which user of the plurality of users is speaking, and enhance the audio or video output of the speaking user on the output device.

First claim

What is claimed is:

1. A computing device, comprising:
a processor that is configured to communicate with an audio output device and a display to provide a video conferencing session having video and sound output, the processor to
receive i) from a first computing device, streaming audio from a first microphone of the first computing device and video from a first camera of the first computing device, and ii) from a second computing device, streaming audio from a second microphone of the second computing device and video from a second camera of the second computing device;
produce, during the video conferencing session, i) video on the display that includes simultaneous images of a plurality of persons including a first user of the first camera and the first computing device and a second user of the second camera and the second computing device, and ii) sound output by the audio output device that includes audio of the plurality of persons including audio from the first computing device and audio from the second computing device,
determine that there are multiple persons in the video conferencing session, and
determine when at least one of the multiple persons is speaking, and in response enhance audio and video of said at least one of the multiple persons, relative to audio and video of remaining one or more of the multiple persons.

2. The computing device of claim 1 wherein the processor is configured to enhance video by minimizing or hiding the image of the remaining one or more of the multiple persons.

3. The computing device of claim 2 wherein each person of the multiple persons is in a separate chat window provided by a single video chat program, and the chat window of a speaking person is enhanced.

4. The computing device of claim 3 wherein the chat widow is enhanced by a change in color, inclusion of a border, modification of the border, enlarging the chat window, or pulling the chat window to front on the display.

5. The computing device of claim 4 wherein the processor determines there are multiple persons in the video conferencing session by performing facial recognition on the streaming video from the first camera and on the streaming video from the second camera, or by performing voice recognition on the streaming audio from the first computing device and on the streaming audio from the second computing device.

6. The computing device of claim 1 wherein each person of the multiple persons is in a separate chat window provided by a single video chat program, and the chat window of a speaking person is enhanced.

7. The computing device of claim 6 wherein the chat widow is enhanced by a change in color, inclusion of a border, modification of the border, enlarging the chat window, or pulling the chat window to front on the display.

8. The computing device of claim 7 wherein the processor determines there are multiple persons in the video conferencing session by performing facial recognition on the streaming video from the first camera and on the streaming video from the second camera, or by performing voice recognition on the streaming audio from the first computing device and on the streaming audio from the second computing device.

9. The computing device of claim 1 wherein the processor determines there are multiple persons in the video conferencing session by performing facial recognition on the streaming video from the first camera and on the streaming video from the second camera, or by performing voice recognition on the streaming audio from the first computing device and on the streaming audio from the second computing device.

10. The computing device of claim 1 wherein the processor selects one of the multiple persons in the video conferencing session by detecting a viewing user's tap on an image of the selected person, and focuses on the selected person by transmitting a signal to a location of the selected person to process the audio of the selected person to better capture speech of the selected person.

11. A method for providing a video conferencing session having video and sound output, the method comprising:
receiving in a viewing user's computing device i) streaming audio from a first microphone and video from a first camera of a first computing device, and ii) streaming audio from a second microphone and video from a second camera of a second computing device;
producing, during the video conferencing session by a single video chat program running in the viewing user's computing device, i) video on a display that includes simultaneous images of a plurality of persons including a first user of the first camera and the first computing device and a second user of the second camera and the second computing device, and ii) sound output by an audio output device that includes audio of the plurality of persons including audio from the first computing device and audio from the second computing device;
determining that there are multiple persons in the video conferencing session; and
determining when at least one of the multiple persons is speaking, and in response enhancing audio and video of the at least one of the multiple persons who is speaking, relative to remaining one or more of the multiple persons.

12. The method of claim 11 wherein enhancing video of the at least one of the multiple persons who is speaking comprises
minimizing or hiding the image of only the remaining one or more of the multiple persons.

13. The method of claim 12 wherein each person of the multiple persons is in a separate chat window provided by the single video chat program, and the chat window of the at least of the multiple persons who is speaking is enhanced.

14. The method of claim 13 wherein the chat widow is enhanced by a change in color, inclusion of a border, modification of the border, enlarging the chat window, or pulling the chat window to front on the display.

15. The method of claim 14 wherein determining there are multiple persons in the video conferencing session comprises
performing facial recognition on the streaming video from the first camera and on the streaming video from the second camera, or performing voice recognition on the streaming audio from the first computing device and the streaming audio from the second computing device.

16. The method of claim 11 wherein each person of the multiple persons is in a separate chat window provided by the single video chat program, and the chat window of a speaking person is enhanced.

17. The method of claim 16 wherein the chat widow is enhanced by a change in color, inclusion of a border, modification of the border, enlarging the chat window, or pulling the chat window to front on the display.

18. The method of claim 17 wherein determining there are multiple persons in the video conferencing session comprises
performing facial recognition on the streaming video from the first camera and on the streaming video from the second camera, or performing voice recognition on the streaming audio from the first computing device and the streaming audio from the second computing device.

19. The method of claim 11 wherein determining there are multiple persons in the video conferencing session comprise performing facial recognition on the streaming video from the first camera and on the streaming video from the second camera, or performing voice recognition the streaming audio from the first computing device and the streaming audio from the second computing device.

20. The method of claim 11 further comprising selecting one of the multiple persons in th

Assignees

• Apple Inc

Inventors

• Pance Aleksandar
• Bilbrey Brett
• Hadley Darby E
• Johnson Martin E
• Isaac Ronald Nadim

Classifications

• H04N21/4852

  for modifying audio parameters, e.g. switching between mono and stereo · CPC title

• H04S7/303

  Tracking of listener position or orientation · CPC title

• H04N21/439

  Processing of audio elementary streams · CPC title

• H04N21/42201

  biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user (input arrangements for interaction with the human body based on nervous system activity detection G06F3/015) · CPC title

• H04N21/44218

  Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45) · CPC title