Motion mirroring system that incorporates virtual environment constraints

Patent metadata

Field	Value
Publication number	US-11400362-B2
Application number	US-202017037496-A
Country	US
Kind code	B2
Filing date	Sep 29, 2020
Priority date	May 23, 2017
Publication date	Aug 2, 2022
Grant date	Aug 2, 2022

Abstract

A system that mirrors motion of a physical object by displaying a virtual object moving in a virtual environment. The mirroring display may be used for example for feedback, coaching, or for playing virtual games. Motion of the physical object is measured by motion sensors that may for example include an accelerometer, a gyroscope, and a magnetometer. Sensor data is transmitted to a computer that calculates the position and orientation of the physical object and generates a corresponding position and orientation of the virtual object. The computer may correct or adjust the calculations using sensor data redundancies. The virtual environment may include constraints on the position, orientation, or motion of the virtual object. These constraints may be used to compensate for accumulating errors in position and orientation. The system may for example use proportional error feedback to adjust position and orientation based on sensor redundancies and virtual environment constraints.

First claim

What is claimed is:

1. A motion mirroring system that incorporates virtual environment constraints, comprising:
a motion capture element configured to couple with a moveable object located in a real environment, wherein said motion capture element comprises
a sensor configured to capture one or more values associated with an orientation, position, velocity, acceleration, angular velocity, and angular acceleration of said motion capture element;
a first communication interface; and,
a microprocessor coupled with said sensor and said first communication interface,
wherein said microprocessor is configured to
collect data that comprises said one or more values from said sensor;
transmit said data via said first communication interface;

a computer that comprises
a display;
a memory that contains
a virtual environment state comprising a position and orientation of a virtual object in a virtual environment, wherein said virtual object represents said moveable object in said virtual environment,
wherein said virtual environment state mirrors motion of said moveable object in said real environment with motion of said virtual object in said virtual environment;

one or more constraints on said position and orientation of said virtual object in said virtual environment;

a second communication interface configured to communicate with said first communication interface to obtain said data;
wherein said computer is configured to receive said data via said second communication interface;
calculate a position and orientation of said moveable object in said real environment from said data;
apply one or more corrections to said position and orientation of said moveable object in said real environment based on one or more redundancies in said data,
wherein said one or more redundancies in said data is obtained from said sensor;

transform said position and orientation of said moveable object in said real environment into said position and orientation of said virtual object in said virtual environment after said apply said one or more corrections;
apply said one or more constraints in said virtual environment to compensate for accumulating errors in said position or orientation;
determine whether said position and orientation of said virtual object in said virtual environment satisfies said one or more constraints;
when said position and orientation of said virtual object in said virtual environment does not satisfy one or more of said one or more constraints, modify said position and orientation of said virtual object in said virtual environment to satisfy said one or more constraints to compensate for an error in said position and orientation not fully corrected by said one or more corrections as an error correction applied after said one or more constraints are applied in said virtual environment,
wherein said error correction is in addition to said one or more corrections; and

display said virtual environment and said virtual object on said display.

2. The system of claim 1 , wherein said moveable object is one or more of a piece of equipment, an article of clothing, or a body part of a person.

3. The system of claim 1 , wherein said sensor comprises a 3-axis accelerometer, a 3-axis rate gyroscope, and a 3-axis magnetometer.

4. The system of claim 3 , wherein said redundancies in said data comprise
a calculation of said orientation of said moveable object in said real environment by integration of angular velocity data from said 3-axis rate gyroscope;
a measurement of said orientation of said moveable object in said real environment relative to an Earth magnetic field from said 3-axis magnetometer; and
a measurement of said orientation of said moveable object in said real environment relative to an Earth gravitational field from said 3-axis accelerometer, when said data indicates that said moveable object is substantially stationary.

5. The system of claim 4 , wherein
said calculate a position and orientation of said moveable object in said real environment from said data comprises said integration of angular velocity data from said 3-axis rate gyroscope; and,
said apply one or more corrections to said position and orientation of said moveable object in said real environment comprises one or more of
calculate a first rotational error between
a predicted magnetic field vector based on said integration of angular velocity data, and
a measured magnetic field vector from said 3-axis magnetometer;

apply a first fraction of said first rotational error to said orientation of said moveable object in said real environment;
calculate a second rotational error between
a predicted gravitational field vector based on said integration of angular velocity data, and
a measured gravitational field vector from said 3-axis accelerometer;

apply a second fraction of said second rotational error to said orientation of said moveable object in said real environment.

6. The system of claim 1 , wherein
said virtual environment comprises a virtual camera that forms an image of said virtual environment;
said display said virtual environment and said virtual object on said display transmits said image of said virtual environment to said display; and,
a position and orientation of said virtual camera in said virtual environment is modifiable by a user.

7. The system of claim 1 , wherein a change in said position and orientation of said moveable object in said real environment results in a corresponding change within a predefined time period of said position and orientation of said virtual object in said virtual environment shown on said display.

8. The system of claim 1 , wherein said one or more constraints comprise a region in said virtual environment that said virtual object must be located within.

9. The system of claim 1 , wherein said one or more constraints comprise a location in said virtual environment that said virtual object must be located near.

10. The system of claim 9 , wherein said modify said position and orientation of said virtual object in said virtual environment to satisfy said one or more constraints comprises
calculate an error vector between said position of said virtual object in said virtual environment and said location; and
add a fraction of said error vector to said position of said virtual object in said virtual environment.

11. The system of claim 1 , wherein said one or more constraints comprise one or more of a maximum speed, a minimum speed, a maximum acceleration, a minimum acceleration, a maximum angular velocity, a minimum angular velocity, a maximum angular acceleration, and a minimum angular acceleration of said virtual object.

12. The system of claim 1 , wherein said one or more constraints comprise one or more virtual barriers that said virtual object cannot pass through.

13. The system of claim 12 , wherein said one or more virtual barriers comprise a ground surface.

14. The system of claim 1 , wherein
said moveable object is a piece of sporting equipment used in a sports activity;
said virtual environment is a virtual game for said sports activity; and,
said virtual object in said virtual environment is a representation of said piece of sporting equipment.

15. The system of claim 14 , wherein said sports activity is one or more of golf, tennis, badminton, racquetball, table tennis, squash, baseball, softball, cricket, hockey, field hockey, croquet, football, rugby, Australian rules football, soccer, volleyball, water polo, polo, basketball, lacrosse, billiards, horseshoes, shuffleboard, handball, bocce, bowling, dodgeball, kick ball, track and

Assignees

• Blast Motion Inc

Inventors

• Haas Juergen
• Bose Bhaskar
• Bentley Michael
• Kaps Ryan

Classifications

• A63F2300/8082

  Virtual reality · CPC title

• G06T15/20

  Perspective computation · CPC title

• A63F13/25

  Output arrangements for video game devices · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• A63F13/525

  Changing parameters of virtual cameras · CPC title