Auto-completion for gesture-input in assistant systems

Patent metadata

Field	Value
Publication number	US-11869231-B2
Application number	US-202318150737-A
Country	US
Kind code	B2
Filing date	Jan 5, 2023
Priority date	Apr 20, 2018
Publication date	Jan 9, 2024
Grant date	Jan 9, 2024

Abstract

A method includes detecting a user input comprising an incomplete three-dimensional (3D) gesture performed by one or more hands of a first user by a virtual-reality (VR) headset, selecting candidate 3D gestures from pre-defined 3D gestures based on a personalized gesture-recognition model, wherein each of the candidate 3D gestures is associated with a confidence score representing a likelihood the first user intended to input the respective candidate 3D gesture, and presenting one or more suggested inputs corresponding to one or more of the candidate 3D gestures at the VR headset.

First claim

What is claimed is:

1. A method comprising, by a virtual-reality (VR) headset:
detecting a user input comprising an incomplete three-dimensional (3D) gesture in the air performed by one or more hands of a first user;
selecting, based on a personalized gesture-recognition model, one or more candidate 3D gestures from a plurality of pre-defined 3D gestures, wherein each of the candidate 3D gestures is associated with a confidence score representing a likelihood the first user intended to input the respective candidate 3D gesture by performing the incomplete 3D gesture in the air; and
presenting, at the VR headset, one or more suggested inputs corresponding to one or more of the candidate 3D gestures.

2. The method of claim 1 , further comprising:
calculating, by the VR headset for each of the one or more candidate 3D gestures, a similarity level of the candidate 3D gesture with respect to the incomplete 3D gesture.

3. The method of claim 2 , wherein the similarly level of each candidate 3D gesture with respect to the incomplete 3D gesture is based on a trajectory of the incomplete 3D gesture with respect to the VR headset.

4. The method of claim 2 , wherein the similarly level of each candidate 3D gesture with respect to the incomplete 3D gesture is based on an orientation of the incomplete 3D gesture with respect to the VR headset.

5. The method of claim 2 , wherein the similarly level of each candidate 3D gesture with respect to the incomplete 3D gesture is based on an object associated with the incomplete 3D gesture.

6. The method of claim 2 , wherein the similarly level of each candidate 3D gesture with respect to the incomplete 3D gesture is based on contextual information associated with the incomplete 3D gesture.

7. The method of claim 2 , wherein the similarly level of each candidate 3D gesture with respect to the incomplete 3D gesture is based on a position of the incomplete 3D gesture with respect to the VR headset.

8. The method of claim 1 , further comprising:
calculating, by the VR headset, one or more confidence scores for one or more intents corresponding to the incomplete 3D gesture; and
determining, by the VR headset, that each of the one or more confidence scores is below a threshold score.

9. The method of claim 8 , wherein the threshold score is based on a 3D wake-up gesture performed by the first user.

10. The method of claim 8 , wherein calculating the one or more confidence scores for the one or more intents corresponding to the incomplete 3D gesture is based on a velocity associated with the incomplete 3D gesture.

11. The method of claim 8 , wherein calculating the one or more confidence scores for the one or more intents corresponding to the incomplete 3D gesture is based on temporal information associated with the incomplete 3D gesture, and wherein the temporal information comprises a pause in the user input.

12. The method of claim 8 , wherein selecting the one or more candidate 3D gestures is further based on the one or more intents.

13. The method of claim 1 , further comprising:
receiving, at the VR headset, a user-selected input from the first user, wherein the user-selected input comprises one of the suggested inputs; and
executing, by the VR headset, one or more tasks based on the user-selected input.

14. The method of claim 1 , wherein each pre-defined 3D gesture comprises one or more of pointing, poking, tapping, waving, or swiping.

15. The method of claim 1 , further comprising:
receiving, at the VR headset, a first user-selected input from the first user, wherein the first user-selected input comprises one of the suggested inputs, and wherein the first user-selected input is associated with a first intent;
generating, by the VR headset based on the first user-selected input, one or more additional candidate 3D gestures, wherein each of the one or more additional candidate 3D gestures is associated with the first intent;
presenting, at the VR headset, one or more additional suggested inputs corresponding to one or more of the additional candidate 3D gestures;
receiving, at the VR headset, a second user-selected input from the first user, wherein the second user-selected input comprises one of the additional suggested inputs; and
executing, by the VR headset, one or more tasks based on the second user-selected input.

16. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
detect, by a virtual-reality (VR) headset, a user input comprising an incomplete three-dimensional (3D) gesture in the air performed by one or more hands of a first user;
select, based on a personalized gesture-recognition model by the VR headset, one or more candidate 3D gestures from a plurality of pre-defined 3D gestures, wherein each of the candidate 3D gestures is associated with a confidence score representing a likelihood the first user intended to input the respective candidate 3D gesture by performing the incomplete 3D gesture in the air; and
present, at the VR headset, one or more suggested inputs corresponding to one or more of the candidate 3D gestures.

17. The media of claim 16 , wherein the software is further operable when executed to:
calculate, by the VR headset for each of the one or more candidate 3D gestures, a similarity level of the candidate 3D gesture with respect to the incomplete 3D gesture.

18. The media of claim 17 , wherein the similarly level of each candidate 3D gesture with respect to the incomplete 3D gesture is based on a trajectory of the incomplete 3D gesture with respect to the VR headset.

19. The media of claim 17 , wherein the similarly level of each candidate 3D gesture with respect to the incomplete 3D gesture is based on an orientation of the incomplete 3D gesture with respect to the VR headset.

20. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
detect, by a virtual-reality (VR) headset, a user input comprising an incomplete three-dimensional (3D) gesture in the air performed by one or more hands of a first user;
select, based on a personalized gesture-recognition model by the VR headset, one or more candidate 3D gestures from a plurality of pre-defined 3D gestures, wherein each of the candidate 3D gestures is associated with a confidence score representing a likelihood the first user intended to input the respective candidate 3D gesture by performing the incomplete 3D gesture in the air; and
present, at the VR headset, one or more suggested inputs corresponding to one or more of the candidate 3D gestures.

Assignees

• Meta Platforms Tech Llc

Inventors

• Presant William Crosby
• Penov Francislav P
• Kumar Anuj

Classifications

• G06Q10/40

  Business processes related to social networking or social networking services · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06Q10/42

  Determination of affinities or common interests between users · CPC title