Context-awareness

Patent metadata

Field	Value
Publication number	US-11244165-B2
Application number	US-202016917480-A
Country	US
Kind code	B2
Filing date	Jun 30, 2020
Priority date	Apr 27, 2018
Publication date	Feb 8, 2022
Grant date	Feb 8, 2022

Abstract

The discussion relates to context-aware environments. One example can include inwardly-facing cameras positioned around a periphery of an environment that defines a volume. The example can also include sensors positioned relative to the volume and configured to communicate with a user device in the volume. The example can also include an ambient perception component configured to track user locations in the volume and to detect user gestures relative to objects in the volume, and responsive to receiving a query from the user's device, to supplement the query with information derived from the objects.

First claim

The invention claimed is:

1. A system, comprising:
cameras positioned to capture portions of an environment;
sensors positioned to sense the portions or other portions of the environment and to communicate with user devices in the environment; and,
a processor configured to determine locations of users in the environment utilizing data from the cameras, to identify the user devices in the environment utilizing data from the sensors, to co-locate individual users and individual user devices, to detect user gestures from the camera data, to receive a user query from an individual user performing an individual user gesture, to match the user query to an individual object in the environment, to supplement the user query with information about the individual object, to obtain a response for the supplemented user query, and to send the response to an individual user device that is co-located with the individual user.

2. The system of claim 1 , wherein the processor is further configured to detect the user gestures from both the camera data and the sensor data.

3. The system of claim 2 , wherein the processor is further configured to utilize data from the user devices to detect the user gestures.

4. The system of claim 1 , wherein the user gestures include what object the individual user is looking at when the user query is received, and wherein the object the individual user is looking at comprises the individual object.

5. The system of claim 4 , wherein the processor is further configured to identify the individual object from the camera data.

6. The system of claim 5 , wherein the processor is further configured to obtain information about the individual object from an object database associated with the environment.

7. The system of claim 6 , wherein in an instance where the user query contains a pronoun, the supplement the user query with the user gestures comprises supplementing the user query with the information about the individual object.

8. A system, comprising:
cameras positioned to capture images of portions of an environment;
sensors positioned to sense the portions or other portions of the environment and configured to communicate with a user device in the environment; and,
an ambient perception component configured to track locations of a user in the environment and to detect user input relative to objects in the environment, and responsive to receiving a user query from the user, to match the user query to an individual object associated with an individual user input, to supplement the user query with information about the individual object, and to provide a response to the supplemented user query to the user.

9. The system of claim 8 , wherein the ambient perception component is further configured to identify that the individual user input comprises a user gesture that is directed to the individual object.

10. The system of claim 9 , wherein the ambient perception component is configured to access a database that relates to the objects and obtain the information about the individual object from the database.

11. The system of claim 10 , wherein the ambient perception component is configured to supplement the user query with at least some of the information about the individual object from the database.

12. The system of claim 10 , wherein the ambient perception component is configured to submit the supplemented user query to a search tool and send the response that includes results from the search tool to the user device for presentation to the user.

13. A smart device, comprising:
storage configured to store computer-readable instructions; and,
a processor configured to execute the computer-readable instructions: to receive a response to a supplemented user query, the supplemented user query comprising a user query submitted by a user co-located with the smart device, the user query matched to an object associated with a user gesture and supplemented with object information relating to the object associated with the user gesture, and to cause the response to the supplemented user query to be presented to the user.

14. The smart device of claim 13 , wherein the computer-readable instructions when executed by the processor cause the processor to audibly present the response.

15. The smart device of claim 13 , further comprising sensors configured to sense the user gesture.

16. The smart device of claim 15 , wherein the sensors comprise a camera.

17. The smart device of claim 15 , wherein the sensors comprise microelectromechanical sensors configured to detect user motions.

18. The smart device of claim 13 , wherein the computer-readable instructions when executed by the processor cause the processor to communicate with other devices in an environment containing the object.

19. The smart device of claim 18 , wherein the computer-readable instructions when executed by the processor cause the processor to receive the user gesture and the object information and to generate the supplemented user query from the object information and the user query.

20. The smart device of claim 18 , wherein the computer-readable instructions when executed by the processor cause the processor to receive the user query and to receive the response to the supplemented user query.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Abhishek Abhishek
• Zuberi Khawar
• Liu Jie
• Aminpour Rouzbeh
• Zhang Zhengyou
• Dalloul Ali
• Lymberopoulos Dimitrios
• Goraczko Michel
• Capone Tony
• Wang Di
• Lu Yi
• Asmi Yasser B

Classifications

• G06F16/903

  Querying (for retrieval from the web G06F16/953) · CPC title

• G06Q30/0205 Primary

  based on location or geographical consideration · CPC title

• G06F16/9535

  Search customisation based on user profiles and personalisation · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title

• G06Q30/0267

  Wireless devices · CPC title