Systems and methods for recording haptic data for use with multi-media data

Patent metadata

Field	Value
Publication number	US-10650859-B2
Application number	US-201715728704-A
Country	US
Kind code	B2
Filing date	Oct 10, 2017
Priority date	Dec 19, 2014
Publication date	May 12, 2020
Grant date	May 12, 2020

Abstract

A system includes a recorder configured to record audio and/or video of a subject of interest and output a recording of the subject of interest and a non-contact sensor associated with the recorder. The non-contact sensor is constructed and arranged to measure movement and/or vibration of the subject of interest from substantially the same perspective and at the same time as the recorder. The system includes a controller configured to transform the measured movement and/or vibration of the subject of interest measured by the non-contact sensor into a tactile data stream for sending to a haptic display device for playback with the recording of the subject of interest by the recorder and providing haptic effects corresponding to the measured movement and/or vibration to a user of the haptic display device in synchronization with the recording.

First claim

What is claimed is:

1. A controller, comprising:
an audio/video (A/V) data encoder configured to encode audio or video data of a subject of interest into an audio or video signal, the audio or video data being generated from a recording of the subject of interest by a recorder;
a sensor data encoder configured to encode sensor data to generate encoded sensor data, the sensor data being generated based on a movement or a vibration of the subject of interest during the recording and being obtained by a first sensor not in contact with the subject of interest and tracking the subject of interest and a second sensor located on the subject of interest and tracking the subject of interest, wherein the sensor data includes non-contact sensor data obtained by the first sensor and on-subject sensor data obtained by the second sensor;
an output transformer configured to generate a tactile effect data signal from the encoded sensor data by mixing the non-contact sensor data and the on-subject sensor data, the tactile effect data signal including haptic effects corresponding to the movement or the vibration of the subject of interest during the recording; and
a transmitter configured to transmit the tactile effect data signal, along with the audio or video signal, to a plurality of devices for playback of the haptic effects in synchronization with the recording.

2. The controller of claim 1 , wherein the transmitter is configured to broadcast the tactile effect data signal, along with the audio or video signal, to the plurality of devices for an event in real time.

3. The controller of claim 1 , wherein the first sensor is one selected from at least one of a laser vibrometer and an optical sensor.

4. The controller of claim 1 being configured to control a movement of the recorder using a first servomechanism, and a movement of the first sensor using a second servomechanism.

5. The controller of claim of 4 , wherein the movement of the first sensor is synchronized by the second servomechanism with the movement of the recorder.

6. The controller of claim 4 , wherein the second servomechanism is configured to move the first sensor to a position or an orientation having a same perspective as the recorder.

7. The controller of claim 1 , wherein
the on-subject sensor data has a different frequency, a different sensitivity or a different sampling rate than the non-contact sensor data, and
the controller further comprises:
an input transformer configured to transform the encoded sensor data into a single data stream having a same frequency, a same sensitivity or a same rate.

8. The controller of claim 7 , wherein
the first sensor is at least one of a laser vibrometer and an optical sensor, and
the second sensor is at least one of a tracking device and an accelerometer.

9. A method of providing haptics, comprising:
recording an audio or a video of a subject of interest with a recorder;
tracking the subject of interest with a first sensor not in contact with the subject of interest to generate non-contact sensor data based on a movement or a vibration of the subject of interest during the audio or the video and a second sensor located on the subject of interest to generate on-subject sensor data based on the movement or the vibration, wherein a movement of the first sensor is synchronized with a movement of the recorder;
mixing the non-contact sensor data and the on-subject sensor data to generate sensor data; and
generating tactile effect data from the sensor data, the tactile effect data including haptic effects corresponding to the movement or the vibration of the subject of interest during the audio or the video; and
transmitting the tactile effect data, along with the audio or the video, to a plurality of devices for playback of the haptic effects in synchronization with the audio or the video.

10. The method of claim 9 , further comprising:
controlling the movement of the recorder with a first servomechanism, and
controlling the movement of the first sensor with a second servomechanism, the first and second servomechanism being controlled by a single controller,
wherein generating the tactile effect data includes generating the haptic effects using the single controller.

11. The method of claim 9 , wherein tracking the subject of interest with the first sensor includes
moving the recorder towards the subject of interest, and
moving the first sensor to a position or an orientation having a same perspective as the recorder.

12. The method of claim 9 , wherein the first sensor is at least one of a laser vibrometer or an optical sensor.

13. The method of claim 9 , wherein transmitting the tactile effect data includes broadcasting the tactile effect data, along with the audio or video, to the plurality of devices for an event in real time.

14. The method of claim 9 , further comprising:
tracking the subject of interest with the first sensor and a second sensor,
wherein the non-contact sensor data has a different sampling frequency, a different sensitivity or a different sampling rate than the on-subject sensor data, and
wherein generating tactile effect data from the sensor data includes combining the non-contact sensor data and the on-subject sensor data into a single data stream having a same frequency, a same sensitivity or a same rate.

15. The method of claim 14 , wherein
the first sensor is a at least one of a laser vibrometer and an optical sensor, and
the second sensor is at least one of a tracking device and an accelerometer.

16. A non-transitory computer readable medium having instructions stored thereon that, when executed by a processor, cause the processor to:
record an audio or a video of a subject of interest with a recorder;
track the subject of interest with a first sensor not in contact with the subject of interest to generate non-contact sensor data based on a movement or a vibration of the subject of interest during the audio or the video and a second sensor located on the subject of interest to generate on-subject sensor data based on the movement or the vibration, wherein a movement of the first sensor is synchronized with a movement of the recorder; and
generate tactile effect data from the sensor data, the tactile effect data including haptic effects corresponding to the movement or the vibration of the subject of interest during the audio or the video; and
transmit the tactile effect data, along with the audio or the video, to a plurality of devices for playback of the haptic effects in synchronization with the audio or the video;
an output transformer configured to generate a tactile effect data signal from the encoded sensor data by mixing the non-contact sensor data and the on-subject sensor data, the tactile effect data signal including haptic effects corresponding to the movement or the vibration of the subject of interest during the recording; and
a transmitter configured to transmit the tactile effect data signal, along with the audio or video signal, to a plurality of devices for playback of the haptic effects in synchronization with the recording.

17. The non-transitory computer readable medium of claim 16 , wherein the instructions, when executed by the processor, further cause the processor to:
control the movement of the recorder with a first servomechanism, and
control the movement of the first sensor with a second servomechanism, the first and second servomechanism being controlled by a single controller,
wherein generating the tactile effect data includes generating the haptic effects using the single controller.

Assignees

• Immersion Corp

Inventors

• Ullrich Christopher J
• Lacroix Robert

Classifications

• H04N21/42201

  biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user (input arrangements for interaction with the human body based on nervous system activity detection G06F3/015) · CPC title

• G06F3/016

  Input arrangements with force or tactile feedback as computer generated output to the user · CPC title

• H04N9/8205

  involving the multiplexing of an additional signal and the colour video signal · CPC title

• H04N5/77

  between a recording apparatus and a television camera · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title