Methods and systems for six degree-of-freedom haptic interaction with streaming point data

Patent metadata

Field	Value
Publication number	US-9477307-B2
Application number	US-201414164114-A
Country	US
Kind code	B2
Filing date	Jan 24, 2014
Priority date	Jan 24, 2013
Publication date	Oct 25, 2016
Grant date	Oct 25, 2016

Abstract

Methods, articles of manufacture, and devices related to generating six degree of freedom (DOF) haptic feedback are provided. A computing device can receive first depth data about an environment. The computing device can generate a first plurality of points from the first depth data. The computing device can determine a virtual tool, where the virtual tool is specified in terms of a translation component for the virtual tool and a rotation component for the virtual tool. The computing device can determine a first force vector between the virtual tool and the first plurality of points. The computing device can send a first indication of haptic feedback based on the first force vector.

First claim

The invention claimed is:

1. A method, comprising:
receiving first depth data about an environment at a computing device;
generating a first plurality of points from the first depth data using the computing device, wherein the first depth data comprise depth data about a target object in the environment;
determining a virtual tool using the computing device, wherein the virtual tool is specified in terms of a translation component for the virtual tool and a rotation component for the virtual tool;
determining a virtual fixture providing an overlay of sensory information using the computing device, wherein the virtual fixture is associated with one or more points related to the target object, and wherein the virtual fixture comprises one or more of:
a guidance fixture configured to define a guidance region represented by the depth data about the target object where the virtual tool is permitted to enter, and
a forbidden-region fixture configured to define a forbidden region represented by the depth data about the target object where the virtual tool is not permitted to enter;

determining a first force vector between the virtual tool and the virtual fixture using the computing device by at least:
determining a bounding box for the virtual tool based on a projection of the virtual tool onto the first depth data, and
determining a number of neighboring points of the first plurality of points, wherein each neighboring point is within the bounding box; and

sending, from the computing device, a first indication of haptic feedback based on the first force vector.

2. The method of claim 1 , wherein the virtual fixture comprises a guidance fixture,
and wherein determining the first force vector comprises:
determining whether an interface point associated with the virtual tool is within the guidance region of the guidance fixture, and

after determining that the interface point is within the guidance region, determining that the first force vector comprises a force vector toward the target object.

3. The method of claim 1 , wherein the virtual fixture comprises a forbidden-region fixture, and wherein determining the first force vector comprises:
determining whether an interface point associated with the virtual tool is near or within the forbidden region of the forbidden-region fixture, and
after determining that the interface point is within the forbidden region, determining that the first force vector comprises a force vector away from the target object.

4. The method of claim 1 , wherein the computing device is configured to communicate with a controllable mechanism, wherein sending the indication of haptic feedback based on the force vector from the computing device comprises sending the indication of haptic feedback based on the force vector from the computing device to the controllable mechanism, and wherein the method further comprises:
providing haptic feedback based on the indication of haptic feedback using the controllable mechanism.

5. The method of claim 4 , wherein determining the virtual fixture comprises:
receiving, at the computing device, input specifying one or more boundaries of the virtual fixture using the controllable mechanism; and
designating the virtual fixture based on the input specifying one or more boundaries of the virtual fixture.

6. The method of claim 1 , wherein the first depth data comprises first image data and corresponding first depth values, and wherein the method further comprises:
generating a virtual environment based on the first image data and corresponding first depth values.

7. The method of claim 6 , wherein the virtual environment comprises a first object related to the target object, and wherein the method further comprises:
determining a contact between the first object and the virtual tool.

8. The method of claim 1 , wherein the first depth data comprises data about an input plurality of points in the environment, and wherein determining the first plurality of points comprises:
generating a filtered plurality of points by filtering the input plurality of points using a bilateral filter; and
determining a normal vector for each point in the filtered plurality of points.

9. The method of claim 8 , wherein the computing device comprises a graphics processing unit (GPU), and wherein determining the normal vector for each point in the filtered plurality of points comprises determining the normal vector for each point in the filtered plurality of points using the GPU.

10. The method of claim 1 , wherein determining the virtual tool comprises representing the virtual tool using a plurality of voxels, wherein each voxel has a voxel size in each of three dimensions, and wherein the voxel size for each of the three dimensions is a same size.

11. The method of claim 1 , wherein the environment comprises an underwater environment.

12. The method of claim 1 , wherein the method further comprises:
determining whether the number of neighboring points of the first plurality of points indicates either there are zero neighboring points of the first plurality of points within the bounding box or more than zero neighboring points of the first plurality of points within the bounding box; and
in response to determining that the number of neighboring points of the first plurality of points indicates that there are zero neighboring points of the first plurality of points within the bounding box:
determining that the virtual tool is in a free-motion state, and
moving the virtual tool in a direction and a distance, wherein the direction is based on the translation component, and wherein the distance is bounded by a voxel size.

13. The method of claim 12 , wherein the method further comprises:
in response to determining that the number of neighboring points of the first plurality of points indicates that there are more than zero neighboring points of the first plurality of points within the bounding box:
determining a depth for each neighboring point with respect to the virtual tool;
determining in-contact neighboring points from the neighboring points of the first plurality of points within the bounding box, wherein each in-contact neighboring point has a corresponding depth of zero;
determining in-collision neighboring points from the neighboring points of the first plurality of points within the bounding box, wherein each in-collision neighboring point has a corresponding depth less than zero;
in response to determining that no in-collision neighboring points are determined and to determining at least one in-contact neighboring point is determined:
determining a first constrained acceleration for the virtual tool based on the at least one in-contact neighboring point, and
moving the virtual tool in a direction based on the first constrained acceleration.

14. The method of claim 13 , wherein the method further comprises:
in response to determining that at least one in-collision neighboring point is determined:
determining a second constrained acceleration for the virtual tool based on the at least one in-collision neighboring point; and
moving the virtual tool in a direction based on the second constrained acceleration.

15. The method of claim 1 , wherein the virtual fixture comprises a forbidden-region fixture, and wherein the first indication of haptic feedback comprises an indication of haptic feedback indicating a slowing of movement within the forbidden region.

16. The method of claim 1 , further comprising:
controlling a motion of at least one virtual object within a virtual environment, wherein the virtual environm

Assignees

• Univ Washington Through Its Center For Commercialization
• Univ Washington

Inventors

• Chizeck Howard Jay
• Ryden Fredrik

Classifications

• H04N23/56

  provided with illuminating means · CPC title

• G06F3/016 Primary

  Input arrangements with force or tactile feedback as computer generated output to the user · CPC title

• H04N5/2256

  Electricity · mapped topic

• A61B34/25

  User interfaces for surgical systems · CPC title

• G06T15/04

  Texture mapping · CPC title