Touch and social cues as inputs into a computer

Patent metadata

Field	Value
Publication number	US-9536350-B2
Application number	US-201213689471-A
Country	US
Kind code	B2
Filing date	Nov 29, 2012
Priority date	Aug 24, 2011
Publication date	Jan 3, 2017
Grant date	Jan 3, 2017

Abstract

A system for automatically displaying virtual objects within a mixed reality environment is described. In some embodiments, a see-through head-mounted display device (HMD) identifies a real object (e.g., a person or book) within a field of view of the HMD, detects one or more interactions associated with real object, and automatically displays virtual objects associated with the real object if the one or more interactions involve touching or satisfy one or more social rules stored in a social rules database. The one or more social rules may be used to infer a particular social relationship by considering the distance to another person, the type of environment (e.g., at home or work), and particular physical interactions (e.g., handshakes or hugs). The virtual objects displayed on the HMD may depend on the particular social relationship inferred (e.g., a friend or acquaintance).

First claim

What is claimed is:

1. A method for automatically displaying virtual objects on a mobile device, comprising:
receiving one or more images associated with a field of view of the mobile device;
identifying a particular object located within the field of view using the one or more images, the particular object comprises a particular person;
detecting one or more interactions between a person associated with the mobile device and the particular person;
acquiring an electronic calendar for the person, the electronic calendar includes an electronically scheduled meeting between the person and the particular person;
determining whether the one or more interactions satisfy at least one social rule of one or more social rules, the at least one social rule is satisfied if the person associated with the mobile device touches the particular person at a point in time coinciding with the electronically scheduled meeting between the person associated with the mobile device and the particular person, wherein the at least one social rule is satisfied if the person performs a handshake or high-five gesture with the particular person at the point in time coinciding with the electronically scheduled meeting between the person and the particular person;
acquiring virtual data associated with an augmented reality environment displayed to the particular person in response to detecting that the person associated with the mobile device has touched the particular person at the point in time coinciding with the electronically scheduled meeting between the person associated with the mobile device and the particular person; and
displaying the virtual data on the mobile device.

2. The method of claim 1 , wherein:
the acquiring virtual data includes a first virtual object.

3. The method of claim 1 , wherein:
the one or more interactions include the particular person speaking in the direction of the mobile device.

4. The method of claim 3 , wherein:
the at least one social rule is satisfied if the person associated with the mobile device touches the particular person and the particular person is located within a particular distance of the mobile device for more than a particular period of time.

5. The method of claim 3 , wherein:
the at least one social rule is satisfied if the particular person is associated with a particular privacy level; and
the determining whether the one or more interactions satisfy at least one social rule includes requesting profile information associated with the particular person, the profile information includes the particular privacy level.

6. The method of claim 3 , wherein:
the detecting one or more interactions includes performing skeletal tracking; and
the one or more interactions include the person associated with the mobile device touching the particular person via a handshake, the mobile device comprises a head-mounted display device.

7. The method of claim 6 , wherein:
the virtual data includes a text description displayed to the particular person.

8. An electronic device for automatically displaying virtual objects, comprising:
a memory, configured to store at least one rule of one or more rules;
one or more processors, configured to receive one or more images associated with a field of view of the electronic device, and identify a particular object located within the field of view using the one or more images, the particular object comprises a particular person, the one or more processors configured to detect one or more interactions between a person associated with the electronic device and the particular person, the one or more processors configured to acquire an electronic calendar for the person, the electronic calendar includes an electronically scheduled meeting between the person and the particular person, the one or more processors configured to determine whether the one or more interactions satisfy the at least one rule, the at least one rule is satisfied if the person associated with the electronic device touches the particular person at a point in time coinciding with the electronically scheduled meeting between the person associated with the electronic device and the particular person, wherein the at least one rule is satisfied if the person performs a handshake or a high-five gesture with the particular person at the point in time coinciding with the electronically scheduled meeting, the one or more processors configured to acquire virtual data associated with an augmented reality environment displayed to the particular person in response to detecting that the person associated with the electronic device has touched the particular person at the point in time coinciding with the electronically scheduled meeting between the person associated with the electronic device and the particular person; and
a display, configured to display the virtual data if the one or more interactions satisfy the at least one rule.

9. The electronic device of claim 8 , wherein:
the one or more interactions include the particular person speaking in the direction of the electronic device.

10. The electronic device of claim 9 , wherein:
the at least one rule is satisfied if the person associated with the electronic device touches the particular person and the particular person is located within a particular distance of the electronic device for more than a particular period of time.

11. The electronic device of claim 9 , wherein:
the at least one rule is satisfied if the particular person is associated with a particular privacy level.

12. The electronic device of claim 9 , wherein:
the one or more processors identify the particular object located within the field of view by performing facial recognition on the particular person.

13. The electronic device of claim 12 , wherein:
the virtual data includes a text description displayed to the particular person.

14. One or more hardware storage devices containing processor readable code for programming one or more processors to perform a method for automatically displaying virtual objects comprising the steps of:
receiving one or more images associated with a field of view of the mobile device;
identifying a particular object located within the field of view using the one or more images, the particular object comprises a particular person;
detecting one or more interactions between a person associated with the mobile device and the particular person;
acquiring an electronic calendar for the person, the electronic calendar includes an electronically scheduled meeting between the person and the particular person;
determining whether the one or more interactions satisfy at least one social rule of one or more social rules, the at least one social rule is satisfied if the person associated with the mobile device touches the particular person at a point in time coinciding with the electronically scheduled meeting between the person associated with the mobile device and the particular person, wherein the at least one social rule is satisfied if the person performs a handshake or high-five gesture with the particular person at the point in time coinciding with the electronically scheduled meeting between the person and the particular person;
acquiring virtual data associated with an augmented reality environment displayed to the particular person in response to detecting that the person associated with the mobile device has touched the particular person at the point in time coinciding with the electronically scheduled meeting between the person associated with the mobile device and the particular person; and
determining if the virtual data has been displayed on the mob

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Novak Christopher Michael
• Liu James
• Latta Stephen
• Andrews Anton O A
• Maitlen Craig R
• Martin Sheridan

Classifications

• G06Q10/1093

  Calendar-based scheduling for persons or groups · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title