Systems and methods for facilitating accessible virtual education

Patent metadata

Field	Value
Publication number	US-11307667-B2
Application number	US-202016892248-A
Country	US
Kind code	B2
Filing date	Jun 3, 2020
Priority date	Jun 3, 2019
Publication date	Apr 19, 2022
Grant date	Apr 19, 2022

Abstract

Various embodiments of systems and methods for a learning environment for accessible virtual computer science education are disclosed.

First claim

What is claimed is:

1. A method, comprising,
generating an expert reach set for a gesture as performed by a tutor wherein the expert reach set defines correctness for transient movement dynamics and hand shape for the gesture;
receiving a video feed of a user performing the gesture;
converting the gesture as performed by the user into a spatio-temporal evolution of continuous variables by:
generating an X and Y coordinate time series, wherein the positional X and Y coordinate time series is representative of positions of a right wrist and a left wrist of the user; and
extracting a hand shape of the user; and
determining a deviation of the gesture as performed by the user from the expert reach set for the gesture; and

providing correctness feedback to the user based on the deviation of the gesture as performed by the user from the expert reach set for the gesture,
wherein the expert reach set is a set of continuous states observed from simulating a hybrid system representative of the gesture over time for a bounded set of initial conditions
wherein the hybrid system is determined by:
segmenting the gesture as captured by the video feed of the user into a plurality of discrete modes:
clustering each mode of the plurality of discrete modes in accordance with their respective triggering mechanism;
obtaining one or more kinematic equations representative of the gesture by:
processing the gesture data using a multi-variate polynomial regression model or Fisher Information with Cramer Rao bound to obtain a set of flow parameters representative of mode transitions; and
clustering the gesture data using a density based approach to the set of flow parameters for each mode of the plurality of modes; and

deriving guard conditions for each cluster of the gesture.

2. The method of claim 1 , wherein the step of generating an expert reach set for a gesture further comprises:
deriving a plurality of initial conditions for the gesture as performed by the tutor by performing a statistical analysis of the tutor's speed, initial positions, and kinematic model parameters; and
performing a reachability analysis using the hybrid system representative of the gesture as performed by the tutor.

3. The method of claim 1 , wherein the X and Y coordinate time series is representative of position of the user's wrists over time and wherein the X and Y coordinate time series is extracted using a plurality of location buckets defined around a face and a chest of the user, as captured by the video feed of the user.

4. The method of claim 1 , further comprising:
training a neural network having a plurality of layers using gesture hand shapes;
processing the hand shape of the tutor and the hand shape of the user using the neural network; and
comparing respective outputs of a penultimate layer of the plurality of layers using a Euclidean distance metric, wherein each output is representative of the hand shape of the tutor and the hand shape of the user.

Assignees

• Banerjee Ayan
• Gupta Sandeep
• Paudyal Prajwal
• Univ Arizona State

Inventors

• Banerjee Ayan
• Gupta Sandeep
• Paudyal Prajwal

Classifications

• G06V20/41

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/23

  Clustering techniques · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title