Digital assistant voice input integration

Patent metadata

Field	Value
Publication number	US-11915696-B2
Application number	US-202117379777-A
Country	US
Kind code	B2
Filing date	Jul 19, 2021
Priority date	Dec 16, 2014
Publication date	Feb 27, 2024
Grant date	Feb 27, 2024

Abstract

A digital assistant supported on devices such as smartphones, tablets, personal computers, game consoles, etc. includes an extensibility client that exposes an interface and service that enables third party applications to be integrated with the digital assistant so the application user experiences are rendered using the native voice of the digital assistant. Specific voice inputs associated with a given application may be registered by developers using a manifest that is loaded when the application is launched on the device so that voice inputs from the device user can be mapped by the digital assistant extensibility client to the appropriate application as input events for consumption. In typical implementations, the manifest is arranged as a declarative document that streamlines application development and provides a seamless user experience by enabling customization of third party applications to integrate the digital assistant's voice and behaviors within the user experience of the application's domain.

First claim

What is claimed:

1. A method for implementing extensibility of a digital assistant operating on a computing device to one or more applications in a runtime environment, comprising:
exposing an interface for receiving application-specific voice commands from manifests associated with respective ones of the applications at installation of the applications, the installations including requests by the applications to an operating system instantiated on the computing device to access resources implemented by the digital assistant;
configuring a user interface to receive voice commands from a computing device user during runtime of the applications;
mapping the voice commands received at the user interface to respective ones of the applications according to the manifests during runtime of the applications; and
forwarding the voice commands to the applications for handling during runtime of the applications in response to the mapping.

2. The method of claim 1 further including rendering user experiences supported by the applications using a voice associated with the digital assistant so that user experiences across the applications utilize one voice.

3. The method of claim 2 further including surfacing options to the computing device user for controlling characteristics of the one voice, the characteristics including one of language, gender associated with the one voice, or accent associated with the one voice.

4. The method of claim 1 further including using contextual data when performing the voice command mapping.

5. The method of claim 4 in which the contextual data comprises one or more of time, date, location of the user, location of the computing device, language, schedule, applications installed on the computing device, user preferences, user behaviors, user activities, stored contacts, call history, messaging history, browsing history, computing device type, computing device capabilities, or communication network type.

6. The method of claim 1 further including providing services to the applications, the services including one or more of language services, vocabulary services, voice services, or synthesized text to speech services.

7. The method of claim 6 in which the voice services are arranged to enable the applications to switch among different voices when rendering the user experiences.

8. The method of claim 6 further including receiving portions of the services from a remote service provider.

9. The method of claim 8 further including supporting the interface with an extensibility client that is configured for interaction with the remote service provider.

10. A computing device, comprising:
one or more processors;
a user interface (UI) for interacting with a user of the computing device using audio or graphics; and
a memory device storing computer-readable instructions which, when executed by the one or more processors, instantiate an operating system (OS) and cause the computing device to
load a manifest of commands at installation of an application, the installation including a request by the application to the OS to access resources implemented by a digital assistant that is operable on the computing device,
execute the application on the computing device,
provide a digital assistant extensibility interface that extends to the executing application during runtime of the application, the resources implemented by the digital assistant,
receive an input from the user at the UI during runtime of the application,
parse the input to identify commands in the manifest during runtime of the application, and
in response to identified commands in the manifest, notify the application of the user input during runtime of the application.

11. The computing device of claim 10 in which the instructions further cause the computing device to pass at least a portion of the user input to the application as an input event to be handled by the application.

12. The computing device of claim 10 in which the instructions further cause the computing device to call a function supported by the application in response to the identified command.

13. The computing device of claim 12 in which the application function is at least partially executed through interactions with the digital assistant using the digital assistant extensibility interface.

14. The computing device of claim 13 in which the digital assistant extensibility interface provides services to the application including one or more of language services, vocabulary services, voice services, or text-to-speech services.

15. The computing device of claim 12 in which the application function is at least partially implemented using the digital assistant to communicate with the user over the UI using voice.

16. The computing device of claim 10 in which the manifest is expressed using declarative code.

17. One or more non-transitory computer-readable storage devices storing instructions which, when executed by one or more processors disposed in a computing device, instantiate an operating system (OS) and cause the computing device to:
instantiate a runtime environment on the computing device in which a digital assistant extensibility client is executable and in which a digital assistant is operable on the computing device;
provide an application programming interface (API) for the digital assistant extensibility client that is exposed during runtime of an application executing in the runtime environment on the computing device;
receive a voice command manifest from the application over the API at installation of the application on the computing device, the installation including a request by the application to the OS to access resources implemented by the digital assistant;
configure the digital assistant extensibility client to listen to voice inputs delivered to the computing device by a computing device user;
map the voice inputs to commands contained in the voice command interface; and
pass the mapped voice inputs to the application over the API during runtime of the application.

18. The one or more non-transitory computer-readable storage devices of claim 17 in which the mapping is performed based on context awareness that is maintained by a digital assistant that is operable on the computing device.

19. The one or more non-transitory computer-readable storage devices of claim 17 in which the executed instructions further cause the computing device to operate the digital assistant to interact with the computing device user via voice.

20. The one or more non-transitory computer-readable storage devices of claim 17 in which the voice input is delivered to the computing device through interactions of the computing device user with a digital assistant that is operable on the computing device.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Liddell Derek
• Zhou Francis
• Yen Cheng-Yi

Classifications

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G10L13/033

  Voice editing, e.g. manipulating the voice of the synthesiser · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G10L2015/227

  of the speaker; Human-factor methodology · CPC title