Virtual reality hand gesture generation

Patent metadata

Field	Value
Publication number	US-2021228978-A1
Application number	US-202117229619-A
Country	US
Kind code	A1
Filing date	Apr 13, 2021
Priority date	Oct 11, 2016
Publication date	Jul 29, 2021
Grant date	—

Abstract

A method including receiving at least one of touch data or force data representing a touch input received at the controller, determining one or more model(s), generating image data using the one or more models, the image data representing at least a hand gesture corresponding to the touch input received at the controller, and transmitting the image data to a virtual reality (VR) environment for display.

First claim

What is claimed is:

1 . A system comprising:
one or more processors; and one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:
receiving, from a handheld controller:
touch data generated by one or more first sensors of the handheld controller, the touch data indicating a touch input received at the handheld controller; and
force data generated by one or more second sensors of the handheld controller, the force data indicating an amount of force associated with the touch input;

selecting a trained model from among a plurality of trained models based at least in part on a characteristic of at least one of the touch data or the force data;
inputting the touch data and the force data to the trained model;
determining, using the trained model, that the touch data and the force data correspond to a hand gesture;
generating image data representing the hand gesture; and
causing an image of the hand gesture to be presented on a display based at least in part on the image data.

2 . The system of claim 1 , wherein the causing the image of the hand gesture to be presented on the display comprises transmitting the image data to a Virtual Reality (VR) headset having the display.

3 . The system of claim 1 , wherein the characteristic comprises at least one of:
a location on the handheld controller where the touch input is received; or the amount of the force associated with the touch input.

4 . The system of claim 1 , wherein the selecting of the trained model from among the plurality of trained models is further based on an additional characteristic of a user holding the handheld controller.

5 . The system of claim 1 , wherein the additional characteristic comprises at least one of:
an age of the user; or a hand size of the user.

6 . The system of claim 1 , wherein the one or more second sensors comprise one or more force sensing resistors (FSRs).

7 . The system of claim 1 , wherein the selecting of the trained model from among the plurality of trained models is further based on comparing at least one of:
the touch data to stored touch data associated with a plurality of trained models; or the force data to stored force data associated with the plurality of trained models.

8 . A method comprising:
receiving, by one or more processors, from a handheld controller, at least one of:
touch data generated by one or more first sensors of the handheld controller, the touch data indicating a proximity of at least a portion of a hand relative to the handheld controller; or
force data generated by one or more second sensors of the handheld controller, the force data indicating an amount of force of a press on the handheld controller;
selecting a trained model from among a plurality of trained models based at least in part on a characteristic of the at least one of the touch data or the force data; inputting, by the one or more processors, the at least one of the touch data or the force data to the trained model; determining, using the trained model, that the at least one of the touch data or the force data corresponds to a hand gesture; generating, by the one or more processors, image data representing the hand gesture; and causing an image of the hand gesture to be presented on a display based at least in part on the image data.

9 . The method of claim 8 , wherein the causing the image of the hand gesture to be presented on the display comprises transmitting the image data to a Virtual Reality (VR) headset having the display.

10 . The method of claim 8 , wherein the characteristic comprises at least one of:
a location on the handheld controller where at least the portion of the hand touched the handheld controller; or the amount of the force of the press.

11 . The method of claim 8 , wherein the selecting of the trained model from among the plurality of trained models is further based on an additional characteristic of a user holding the handheld controller.

12 . The method of claim 8 , wherein the selecting of the trained model from among the plurality of trained models is further based on comparing at least one of:
the touch data to stored touch data associated with the plurality of trained models; or the force data to stored force data associated with the plurality of trained models.

13 . The method of claim 8 , wherein the hand gesture is a first hand gesture, the method further comprising:
predicting a second hand gesture based at least in part on the first hand gesture; generating second image data representing the second hand gesture; receiving, from the handheld controller, after the predicting, at least one of:
second touch data generated by the one or more first sensors of the handheld controller, the second touch data indicating a second proximity of at least the portion of the hand relative to the handheld controller; or
second force data generated by the one or more second sensors of the handheld controller, the second force data indicating a second amount of force of a second press on the handheld controller; and
causing a second image of the second hand gesture to be presented on the display based at least in part on the second image data.

14 . The method of claim 8 , wherein:
the receiving comprises receiving the force data; the one or more second sensors comprise one or more force sensing resistors (FSRs); the characteristic comprises a characteristic of the force data; and the inputting comprises inputting the force data to the trained model.

15 . A system comprising:
one or more processors; and one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:
receiving, from a handheld controller, at least one of:
touch data indicating a proximity of at least a portion of a hand relative to the handheld controller; or
force data indicating an amount of force of a press on the handheld controller;

selecting a trained model from among a plurality of trained models based at least in part on a characteristic of the at least one of the touch data or the force data;
inputting the at least one of the touch data or the force data to the trained model;
determining, using the trained model, that the at least one of the touch data or the force data corresponds to a hand gesture;
generating image data representing the hand gesture; and
causing an image of the hand gesture to be presented on a display based at least in part on the image data.

16 . The system of claim 15 , wherein the image comprises at least one of a three dimensional (3D) representation of the hand gesture or an animation of the hand gesture.

17 . The system of claim 15 , wherein the characteristic comprises at least one of:
a location on the handheld controller where at least portion of the hand touched the handheld controller; or the amount of the force of the press.

18 . The system of claim 15 , wherein the selecting of the trained model from among the plurality of trained models is further based on an additional characteristic of a user holding the handheld controller.

19 . The system of claim 15 , wherein the selecting of the trained model from among the plurality of trained models is further based on comparing at least one of:
the touch data to stored touch data associated w

Assignees

• Valve Corp

Inventors

• Nietfeld Scott Douglas
• Van Den Heuvel Joe

Classifications

• G06F3/0325

  using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image · CPC title

• G06F3/03547

  Touch pads, in which fingers can move on a surface · CPC title

• A63F13/428

  involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes · CPC title

• A63F13/67

  adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use · CPC title

• A63F13/218

  using pressure sensors, e.g. generating a signal proportional to the pressure applied by the player · CPC title