Digital assistant extensibility to third party applications

Patent metadata

Field	Value
Publication number	US-2016335138-A1
Application number	US-201514712364-A
Country	US
Kind code	A1
Filing date	May 14, 2015
Priority date	May 14, 2015
Publication date	Nov 17, 2016
Grant date	—

Abstract

A digital assistant includes an extensibility client that interfaces with application extensions that are built by third-party developers so that various aspects of application user experiences, content, or features may be integrated into the digital assistant and rendered as native digital assistant experiences. Application extensions can use a variety of services provided from cloud-based and/or local sources such as language/vocabulary, user preferences, and context services that add intelligence and contextual relevance while enabling the extensions to plug in and operate seamlessly within the digital assistant context. Application extensions may also access and utilize general digital assistant functions, data structures, and libraries exposed by the services and implement application domain-specific context and behaviors using the programming features captured in the extension. Such extensibility to third party applications can broaden the scope of the database of information that the digital assistant may use to answer questions and perform actions for the user.

First claim

1 . A method for implementing extensibility of a digital assistant operating on a device used by a user to one or more applications, comprising:
operating the digital assistant to maintain context-awareness by monitoring user behaviors and interactions with the device; configuring an interface for interoperating with application-specific services exposed by application extensions associated with respective ones of the applications; receiving input from the device user; using the context-awareness to map the device user input to an application extension for handling; and receiving an application-specific service from the application extension in response to the device user inputs.

2 . The method of claim 1 further including rendering the application-specific service so that user experiences across the applications are exposed to the device user as native digital assistant user experiences and whereby the application-specific service increases a size of a database of answers available to the digital assistant.

3 . The method of claim 1 further including using contextual data when performing the mapping.

4 . The method of claim 3 in which the contextual data comprises one or more of time/date, location of the user or device, language, schedule, applications installed on the device, user preferences, user behaviors, user activities, stored contacts, call history, messaging history, browsing history, device type, device capabilities, or communication network type.

5 . The method of claim 1 further including providing extensibility services to the applications, the extensibility services including one or more of language services, vocabulary services, user preference services, or context services.

6 . The method of claim 5 further including receiving portions of the extensibility services from a remote service provider.

7 . The method of claim 6 further including supporting the interface with an extensibility client that is configured for interaction with the remote service provider.

8 . The method of claim 1 further including loading application-specific resources from a manifest included in the application extension, the application-specific resources at least including keywords that are registered with the digital assistant.

9 . The method of claim 1 in which the application extension further includes logic for implementing a user experience or a user interface using the digital assistant.

10 . The method of claim 1 further including configuring the digital assistant, responsively to voice input, gesture input, or manual input for performing at least one of sharing contact information, sharing a stored contact, scheduling a meeting, looking at a user's calendar, scheduling a reminder, making a call, operating a device, playing games, making purchases, taking notes, scheduling an alarm or wake-up reminder, sending a message, checking social media for updates, scraping a website, interacting with a search service, sharing or showing files, sending a link to a website, or sending a link to a resource.

11 . A device, comprising:
one or more processors; a user interface (UI) for interacting with a user of the device using graphics and audio; and a memory device storing code associated with one or more applications and computer-readable instructions which, when executed by the one or more processors, cause the device to:
expose a digital assistant on the device configured to maintain context-awareness for a device user by monitoring user behaviors and interactions with the device, the digital assistant further configured to interact with the device user using voice interactions through the UI,
receive an input from the device user through the UI,
use the context-awareness to deliver the input to an application extension for handling, the application extension being configured to deliver services from one or more applications that are related to the application extension into user experiences renderable by the digital assistant, and
operate the digital assistant to render the services to the device user through the UI.

12 . The device of claim 11 further including causing the device to expose one or more extensibility services to the application extension.

13 . The device of claim 11 further including causing the device to enable the application extension to load application-specific resources from a manifest into a runtime environment for execution.

14 . The device of claim 13 in which the application extension includes an event handler.

15 . The device of claim 14 in which the application extension includes logic comprising one of script or programming construct.

16 . The device of claim 11 further including causing the device to expose one or more databases associated with the one or more applications that are related to the application extension to the digital assistant using the application extension.

17 . One or more computer-readable memory devices storing instructions which, when executed by one or more processors disposed in a computer server, cause the computer server to:
interoperate with a digital assistant extensibility client on a local device, the digital assistant extensibility client exposing an application programming interface (API) to one or more application extensions being executable on the device, each of the application extensions being configured to deliver services from respective applications into user experiences renderable by a digital assistant; maintain digital assistant extensibility services including at least one of i) language services that enable applications to use one or more different languages when rendering a user experience on the local device, ii) vocabulary services that enable applications to handle unknown words or phrases when rendering the user experience, iii) user preference services that enable applications to employ user preferences that are maintained by the digital assistant, or iv) context services that enable applications to utilize context-awareness when delivering services; and provide the digital assistant extensibility services to the one or more application extensions through the API exposed by the digital assistant extensibility client on the local device
wherein the digital assistant is configured to maintain context-awareness for a device user by monitoring user behaviors and interactions with the device, and
wherein the context-awareness is used to deliver user input to the digital assistant to an application extension.

18 . The one or more computer-readable memory devices of claim 17 in which the digital assistant extensibility services and digital assistant extensibility client provide a platform supporting user experiences that are renderable on the local device across all the applications as a native digital assistant experience.

19 . The one or more computer-readable memory devices of claim 17 in which the application extensions contain application-specific resources written to a manifest which is loaded into a runtime environment.

20 . The one or more computer-readable memory devices of claim 19 in which the application extensions are authored by third party developers.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Surti Tanvi
• Patten Michael
• Lyndersay Sean
• Tong Chee Chen

Classifications

• H04W4/80

  Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication · CPC title

• H04M2250/22

  including a touch pad, a touch sensor or a touch detector · CPC title

• G06F9/44526 Primary

  Plug-ins; Add-ons · CPC title

• H04M2250/74

  with voice recognition means · CPC title

• G06F9/541 Primary

  via adapters, e.g. between incompatible applications · CPC title