Systems, devices, and methods for physical surface tracking with a stylus device in an AR/VR environment

Patent metadata

Field	Value
Publication number	US-11397478-B1
Application number	US-202117186838-A
Country	US
Kind code	B1
Filing date	Feb 26, 2021
Priority date	Feb 26, 2021
Publication date	Jul 26, 2022
Grant date	Jul 26, 2022

Abstract

A method includes receiving sensor data from one or more sensors that corresponds to a location of a stylus device and determining that a tip of the stylus device is in contact with a physical surface based on the sensor data. While in contact, the method includes determining a new location of the tip of the stylus device based on the location data, mitigating a tracking error of the determined new location of the tip of the stylus device by translating the new location from a 3D space domain to a 2D space domain that corresponds to the physical surface and adjusting the new location based on a comparison of the new location and historical locations.

First claim

What is claimed is:

1. A stylus device tracking system comprising:
one or more processors;
a set of one or more sensors controlled by the one or more processors and configured to generate sensor data, the set of one or more sensors selected from the group consisting of: optical sensors and inertial sensors, the sensor data including location data usable to derive a location of a stylus device;
wherein the one or more processors are configured to:
determine that a tip of the stylus device is in contact with a physical surface based on the sensor data;
while the tip of the stylus device is determined to be in contact with the physical surface:
determine a new location of the tip of the stylus device based on the location data;
mitigate a tracking error of the determined new location of the tip of the stylus device based on:
a translation of the new location of the tip of the stylus device from a three-dimensional (3D) space domain to a two-dimensional (2D) space domain that corresponds to the physical surface; and
an adjustment of the new location of the tip of the stylus device in the 2D space based on a comparison of the new location and historical locations,
wherein the stylus device is operable to facilitate a drawing of a virtual line that corresponds to the determined and mitigated new location of a tip of the stylus device, the virtual line comprised of virtual segments formed between the adjusted new location and other new locations, and

apply a smoothing interpolation algorithm that improves an accuracy or continuity of the virtual segments.

2. The stylus device tracking system of claim 1 wherein the smoothing interpolation algorithm is a Catmull-Rom splines implementation.

3. The stylus device tracking system of claim 1 wherein the new location and the historical locations are defined by sample points, and wherein the adjustment of the new location of the tip of the stylus device in the 2D space is based on an application of a mean or median function to a latest n sample points including the new location and the historical locations.

4. The stylus device tracking system of claim 3 wherein the n latest sample points is between 2-10.

5. The stylus device tracking system of claim 1 wherein one or more processors are further configured to:
determine a current, real-time location of the tip of the stylus device based on the sensor data;
generate a temporary segment that extends from the latest new location of the tip of the stylus device to the current, real-time location of the tip of the stylus device.

6. The stylus device tracking system of claim 1 wherein the inertial sensors include an inertial measurement unit (IMU).

7. The stylus device tracking system of claim 1 wherein while the tip of the stylus device is determined not to be in contact with any physical surface including the physical surface, the one or more processors are further configured to apply a different method of mitigating the tracking error of the determined tip of the stylus device.

8. The stylus device tracking system of claim 1 wherein the translation of the new location of the tip of the stylus device from the 3D space domain to the 2D space domain that corresponds to the physical surface occurs for location within a threshold distance from the physical surface.

9. A computer-implemented method of operating a stylus device system, the method comprising:
receiving sensor data from one or more sensors selected from the group consisting of: optical sensors and inertial sensors, the sensor data corresponding to a location of the stylus device;
determining that a tip of the stylus device is in contact with a physical surface based on the sensor data; and
while the tip is determined to be in contact with the physical surface:
determining a new location of the tip of the stylus device based on the location data;
mitigating a tracking error of the determined new location of the tip of the stylus device by:
translating the new location of the tip of the stylus device from a three-dimensional (3D) space domain to a two-dimensional (2D) space domain that corresponds to the physical surface; and
adjusting the new location of the tip of the stylus device in the 2D space based on a comparison of the new location and historical locations,
wherein the stylus device is operable to facilitate a drawing of a virtual line that corresponds to the determined and mitigated new location of a tip of the stylus device, the virtual line comprised of virtual segments formed between each new tracked location of the tip of the stylus device.

10. The method of claim 9 wherein the method further includes:
applying a smoothing interpolation algorithm that improves an accuracy or continuity of the virtual segments between each new tracked location of the tip of the stylus device, wherein the smoothing interpolation algorithm is a Catmull-Rom splines implementation.

11. The method of claim 9 wherein the new location and the historical locations are defined by sample points, and wherein the adjusting of the new location of the tip of the stylus device in the 2D space includes:
applying a mean or median function to a latest n sample points including the new location and the historical locations, wherein determining the new tracked location of the tip of the stylus device is based on a result of the applied mean or median function to the latest n sample points.

12. The method of claim 11 wherein then latest sample points is between 2-10.

13. The method of claim 9 wherein the sensor data includes stylus device pose data that is generated by and received from the optical sensors that include a plurality of receivers or emitters controlled by one or more processors and are configured to facilitate the optical tracking of the location of the stylus device in 3D space.

14. The method of claim 9 wherein the sensor data includes stylus device pose data that is generated by and received from the inertial sensors, the inertial sensors controlled by one or more processors and are configured to facilitate the inertial tracking of the location of the stylus device in 3D.

15. The method of claim 9 further comprising:
determining a current, real-time location of the tip of the stylus device based on the sensor data; and
generating a temporary segment that extends from the latest new tracked location of the tip of the stylus device to the current, real-time location of the tip of the stylus device,
wherein the temporary segment is updated as new locations of the tip of the stylus device are received.

16. The method of claim 9 wherein while the tip is determined not to be in contact with the physical surface, the method does not include translating the new location of the tip of the stylus device from the 3D space domain to the 2D space domain that corresponds to the physical surface.

17. A non-transitory computer-program product tangibly embodied in a machine-readable non-transitory storage medium that includes instructions configured to cause one or more processors to performing operations including:
receiving sensor data from one or more sensors selected from the group consisting of: optical sensors and inertial sensors, the sensor data corresponding to a location of a stylus device;
deriving a location of the stylus device based on the sensor data;
determining that a tip of the stylus device is in contact with a physical surface based on the sensor data; and
while the tip is determined to be in contact with the physical surface:
determining a new location of the tip of the stylus device based o

Assignees

• Logitech Europe Sa

Inventors

• Gutierrez Mario
• Waldron Philip
• Zellweger Fabien
• O'Sullivan Damien

Classifications

• G06F3/03545 Primary

  Pens or stylus · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0325

  using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image · CPC title

• G06F3/038

  Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry · CPC title