Reusable multimodal application

Patent metadata

Field	Value
Publication number	US-12418583-B2
Application number	US-202218068591-A
Country	US
Kind code	B2
Filing date	Dec 20, 2022
Priority date	May 5, 2006
Publication date	Sep 16, 2025
Grant date	Sep 16, 2025

Abstract

A method and system are disclosed herein for accepting multimodal inputs and deriving synchronized and processed information. A reusable multimodal application is provided on the mobile device. A user transmits a multimodal command to the multimodal platform via the mobile network. The one or more modes of communication that are inputted are transmitted to the multimodal platform(s) via the mobile network(s) and thereafter synchronized and processed at the multimodal platform. The synchronized and processed information is transmitted to the multimodal application. If required, the user verifies and appropriately modifies the synchronized and processed information. The verified and modified information are transferred from the multimodal application to the visual application. The final result(s) are derived by inputting the verified and modified results into the visual application.

First claim

The invention claimed is:

1. A method comprising:
receiving, by a mobile computing device, audio input to an application executing on the mobile computing device, the application having an audio mode of communication and a touch input mode of communication;
providing, by the mobile computing device, the audio input to a multimodal platform that is remote from the mobile computing device;
generating, by the multimodal platform, synchronized information by synchronizing the audio input with the touch input mode of communication; and
providing, by the multimodal platform, the synchronized information to the application executing on the mobile computing device,
wherein the synchronized information facilitates the application switching from the audio mode of communication to the touch input mode of communication and providing the synchronized information for verification or modification in the touch input mode of communication while continuing execution of the application.

2. The method of claim 1 further comprising:
generating a search request based on the synchronized information; and transmitting the search request to a remote computer system.

3. The method of claim 2 , wherein the synchronized information comprises a plurality of search terms, and wherein the synchronized information facilitates the mobile computing device rendering the plurality of search terms in the touch input mode of communication and permitting user selection or modification of the plurality of search terms.

4. The method of claim 1 , wherein the application comprises a web browser.

5. The method of claim 1 , wherein the application comprises a navigation application.

6. The method of claim 1 , wherein the mobile computing device is GPS-enabled, and wherein the application accepts GPS input.

7. The method of claim 1 , wherein generating the synchronized information comprises synchronizing a state of the audio input mode with a state of the touch input mode.

8. A mobile computing device comprising a processor and memory, the memory comprising executable instructions configured to, in response to being executed by the processor, cause the mobile computing device to:
receive touch input to an application executing on the mobile computing device, the application having a touch input mode of communication and a second mode of communication;
provide the touch input to a multimodal platform remote from the mobile computing device;
receive synchronized information from the multimodal platform, wherein the synchronized information is generated by the multimodal platform by synchronizing the touch input with the second mode of communication; and
provide the synchronized information to the application executing on the mobile computing device,
wherein the synchronized information facilitates the application switching from the touch input mode of communication to the second mode of communication and providing the synchronized information for verification or modification in the second mode of communication while continuing execution of the application.

9. The mobile computing device of claim 8 , wherein the second mode of communication comprises an audio mode of communication.

10. The mobile computing device of claim 8 further comprising a touchscreen.

11. The mobile computing device of claim 8 , wherein the executable instructions are further configured to cause the mobile computing device to:
generate a search request based on the synchronized information; and transmit the search request to a remote computer system.

12. The mobile computing device of claim 11 , wherein the synchronized information comprises a plurality of search terms, and wherein the synchronized information facilitates the mobile computing device rendering the plurality of search terms in the touch input mode of communication and permitting user selection or modification of the plurality of search terms.

13. The mobile computing device of claim 8 , wherein the application comprises a web browser.

14. The mobile computing device of claim 8 , wherein the application comprises a navigation application.

15. The mobile computing device of claim 8 , wherein the mobile computing device is GPS-enabled, and wherein the application accepts GPS input.

16. A non-transitory computer-readable medium having stored thereon computer-executable instructions configured to, in response to being executed by a computer system comprising a mobile computing device and a multimodal platform remote from the mobile computing device, cause the computer system to:
receive audio input to an application executing on the mobile computing device, the application having an audio mode of communication and a touch input mode of communication;
generate, by the multimodal platform, synchronized information by synchronizing the audio input with the touch input mode of communication; and
provide, by the multimodal platform, the synchronized information to the application executing on the mobile computing device,
wherein the synchronized information facilitates the application switching from the audio mode of communication to the touch input mode of communication and providing the synchronized information for verification or modification in the touch input mode of communication while continuing execution of the application.

17. The non-transitory computer-readable medium of claim 16 , wherein the computer-executable instructions are further configured to cause the computer system to generate a search request based on the synchronized information.

18. The non-transitory computer-readable medium of claim 17 , wherein the synchronized information comprises a plurality of search terms, and wherein the synchronized information facilitates the mobile computing device rendering the plurality of search terms in the touch input mode of communication and permitting user selection or modification of the plurality of search terms.

19. The non-transitory computer-readable medium of claim 16 , wherein the application comprises a web browser.

20. The non-transitory computer-readable medium of claim 16 , wherein the application comprises a navigation application.

21. The non-transitory computer-readable medium of claim 16 , wherein the mobile computing device is GPS-enabled, and wherein the application accepts GPS input.

Assignees

• Gula Consulting Llc

Inventors

• Anderl Ewald C

Classifications

• H04M3/4938

  comprising a voice browser which renders and interprets, e.g. VoiceXML · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G10L15/26

  Speech to text systems (G10L15/08 takes precedence) · CPC title

• G06F16/9538

  Presentation of query results · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title