Digital assistant extensibility to third party applications

Patent metadata

Field	Value
Publication number	US-10133613-B2
Application number	US-201514712364-A
Country	US
Kind code	B2
Filing date	May 14, 2015
Priority date	May 14, 2015
Publication date	Nov 20, 2018
Grant date	Nov 20, 2018

Abstract

A digital assistant includes an extensibility client that interfaces with application extensions that are built by third-party developers so that various aspects of application user experiences, content, or features may be integrated into the digital assistant and rendered as native digital assistant experiences. Application extensions can use a variety of services provided from cloud-based and/or local sources such as language/vocabulary, user preferences, and context services that add intelligence and contextual relevance while enabling the extensions to plug in and operate seamlessly within the digital assistant context. Application extensions may also access and utilize general digital assistant functions, data structures, and libraries exposed by the services and implement application domain-specific context and behaviors using the programming features captured in the extension. Such extensibility to third party applications can broaden the scope of the database of information that the digital assistant may use to answer questions and perform actions for the user.

First claim

What is claimed:

1. A method for implementing extensibility of a digital assistant operating on a device used by a user to one or more applications, comprising:
configuring an interface for interoperating with application-specific services exposed by application extensions associated with respective ones of the applications, wherein each application extension is instantiated locally on the device and comprises a manifest of resources that are specific to a related application, an event handler, and logic;
receiving from the respective ones of the applications a request to access digital assistant resources that describe capabilities and resources to facilitate interaction between the respective ones of the applications and the digital assistant;
loading the manifest of resources from the related application to a digital assistant extensibility client, the manifest of resources including one or more keywords specific to the related application;
registering at least the one or more keywords with the digital assistant extensibility client;
receiving input from the device user to execute an event and invoke the one or more registered keywords;
based on the received input and invocation of the one or more registered keywords, mapping the device user input to an application extension for handling;
receiving an initial application-specific service from the application extension in response to the device user input; and
after receiving the initial application-specific service, receiving a subsequent application-specific service from a different application, in which the subsequent application-specific service is based on context from the initial application-specific service.

2. The method of claim 1 further including rendering the application-specific services so that user experiences across the applications are exposed to the device user as native digital assistant user experiences and whereby the application-specific services increase a size of a database of answers available to the digital assistant.

3. The method of claim 1 further including using contextual data when performing the mapping.

4. The method of claim 3 in which the contextual data comprises one or more of time/date, location of the user or device, language, schedule, applications installed on the device, user preferences, user behaviors, user activities, stored contacts, call history, messaging history, browsing history, device type, device capabilities, or communication network type.

5. The method of claim 1 further including providing extensibility services to the respective ones of the applications, the extensibility services including one or more of language services, vocabulary services, user preference services, or context services.

6. The method of claim 5 further including receiving portions of the extensibility services from a remote service provider.

7. The method of claim 1 in which the application extension further includes logic for implementing a user experience or a user interface using the digital assistant.

8. The method of claim 1 further including configuring the digital assistant, responsively to voice input, gesture input, or manual input for performing at least one of sharing contact information, sharing a stored contact, scheduling a meeting, looking at a user's calendar, scheduling a reminder, making a call, operating a device, playing games, making purchases, taking notes, scheduling an alarm or wake-up reminder, sending a message, checking social media for updates, scraping a website, interacting with a search service, sharing or showing files, sending a link to a website, or sending a link to a resource.

9. The method of claim 1 in which the one or more keywords specifically identify the respective application.

10. The method of claim 1 in which the subsequent application-specific service is received through a series of interactions from the user at the digital assistant.

11. A device, comprising:
one or more processors;
a user interface (UI) for interacting with a user of the device using graphics and audio; and
a memory device storing code associated with one or more applications and computer-readable instructions which, when executed by the one or more processors, cause the device to:
expose a digital assistant on the device configured to maintain context-awareness for a device user by monitoring user behaviors and interactions with the device, the digital assistant further configured to interact with the device user using voice interactions through the UI, wherein each application extension is instantiated locally on the device and comprises a manifest of resources that are specific to a related application, an event handler, and logic,
receive from the respective one or more applications a request to access digital assistant resources that describe capabilities and resources to facilitate interaction between the respective one or more applications and the digital assistant,
load the manifest of resources from the related application to a digital assistant extensibility client, the manifest of resources including one or more keywords specific to the related application,
register at least the one or more keywords with the digital assistant extensibility client,
receive an input from the device user through the UI to execute an event and invoke the one or more registered keywords,
use the context-awareness, received input, and the invocation of the one or more registered keywords to deliver the input to an application extension for handling, the application extension being configured to deliver initial services from a related application,
operate the digital assistant to render the initial services to the device user through the UI, and
after operating the digital assistant to render the initial services associated with an initial application, operate a subsequent service using a subsequent application based on context from the initial services.

12. The device of claim 11 further including instructions to cause the device to expose one or more extensibility services to the application extension.

13. The device of claim 11 further including instructions to cause the device to enable the application extension to load application-specific resources from the manifest into a runtime environment for execution.

14. The device of claim 13 in which the application extension includes the event handler.

15. One or more computer-readable memory devices storing instructions which, when executed by one or more processors disposed in a computer server, cause the computer server to:
interoperate with a digital assistant extensibility client on a local device, the digital assistant extensibility client exposing an application programming interface (API) to one or more application extensions being executable on the local device, each of the application extensions being configured to deliver services from respective applications into user experiences renderable by a digital assistant;
maintain digital assistant extensibility services including i) language services that enable applications to use one or more different languages when rendering a user experience on the local device, or ii) vocabulary services that enable applications to handle unknown words or phrases when rendering the user experience; and
provide the digital assistant extensibility services to the one or more application extensions through the API exposed by the digital assistant extensibility client on the local device,
wherein each application extension is instantiated locally on the local device and comprises a manifest of resources that are specific t

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Surti Tanvi
• Patten Michael
• Lyndersay Sean
• Tong Chee Chen

Classifications

• G06F9/44526 Primary

  Plug-ins; Add-ons · CPC title

• H04W4/80

  Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication · CPC title

• H04M2250/74

  with voice recognition means · CPC title

• G06F9/541 Primary

  via adapters, e.g. between incompatible applications · CPC title

• H04M2250/22

  including a touch pad, a touch sensor or a touch detector · CPC title