Systems and methods for playing virtual music instrument through tracking of fingers with coded light

Patent metadata

Field	Value
Publication number	US-9830894-B1
Application number	US-201615164548-A
Country	US
Kind code	B1
Filing date	May 25, 2016
Priority date	May 25, 2016
Publication date	Nov 28, 2017
Grant date	Nov 28, 2017

Abstract

Finger-tracking systems and methods for virtual instruments playback. The described system tracks the position of user's ten fingers on a projection surface and can be used to play virtual instruments such as virtual piano, drums, and bells. The system tracks the movement of user's ten fingers while keeping them free of encumbrance or excessive postural constraints. More specifically, in one or more embodiments, a coded light based projector is used to send out location signal onto a flat surface, and ten light sensors are mounted on user's fingers to receive these signals and locate user's fingers. Based on their locations and relative distance to a fixed point, a printed music instrument can be used for virtual instrument music playback. With the described tracking system, various embodiments of virtual music instruments may be implemented, including a system and method for virtual piano playing as well as virtual Chinese bell playing on printed keyboard and printed Chinese bell set.

First claim

What is claimed is:

1. A finger-tracking system comprising:
a. a projector configured to project a temporal projector light signal, wherein the temporal projector light signal is encoded, for each pixel of the projector, with an information segment comprising the pixel coordinates of the each pixel of the projector;
b. a plurality of light sensors, each of the plurality of light sensors being attached to a finger of a user, wherein each of the plurality of light sensors is configured to detect the temporal projector light signal and generate a sensor signal; and
c. a processing unit operatively coupled to each of the plurality of light sensors and configured to receive the sensor signal from each of the plurality of light sensors, to determine a location information of each finger of the user and to issue a command based on the detected location of at least one finger of the user.

2. The finger-tracking system of claim 1 , wherein the processing unit determines the location information of each finger of the user by identifying a projector pixel corresponding to the sensor signal from a light sensor attached to the corresponding finger of the user.

3. The finger-tracking system of claim 1 , wherein the issued command causes a sound or a musical note to be synthesized.

4. The finger-tracking system of claim 1 , wherein the location information of each finger of the user is determined in relation to an image of a piano keyboard comprising a plurality of piano keys.

5. The finger-tracking system of claim 4 , wherein the processing unit uses the location information of each finger of the user to determine which piano key of the plurality of piano keys has been pressed.

6. The finger-tracking system of claim 5 , wherein the processing unit causes a sound or a musical note corresponding to pressed piano key to be synthesized.

7. The finger-tracking system of claim 5 , wherein the processing unit causes a sequence of pressed piano keys to be recorded.

8. The finger-tracking system of claim 7 , wherein the recorded sequence of pressed piano keys is compared with a reference sequence to determine a difference and a feedback to the user is generated based on the determined difference.

9. The finger-tracking system of claim 1 , wherein the location information of each finger of the user is determined in relation to an image of a plurality of Chinese bells.

10. The finger-tracking system of claim 9 , wherein the processing unit uses the location information of each finger of the user to determine which bell of the plurality of Chinese bells has been struck.

11. The finger-tracking system of claim 10 , wherein the processing unit causes a sound or a musical note corresponding to the struck bell to be synthesized.

12. The finger-tracking system of claim 1 , wherein the temporal projector light signal projected by the project comprises a plurality of sequential light pulses encoding pixel coordinates of the each pixel of the projector.

13. The finger-tracking system of claim 1 , further comprising a computer system comprising a display unit and operatively coupled with the processing unit and configured to receive from the processing unit the determined location information of each finger of the user and to display the received location information of each finger of the user on the display unit.

14. The finger-tracking system of claim 13 , wherein the location information of each finger of the user is displayed on the display unit in a different color.

15. The finger-tracking system of claim 1 , wherein the projector is a DLP projector.

16. A method for tracking fingers of a user, the method comprising:
a. using a projector to project a temporal projector light signal, wherein the temporal projector light signal is encoded, for each pixel of the projector, with an information segment comprising the pixel coordinates of the each pixel of the projector;
b. detecting the temporal projector light signal using a plurality of light sensors, each of the plurality of light sensors being attached to a finger of a user, wherein each of the plurality of light sensors is configured to detect the temporal projector light signal and generate a sensor signal; and
c. using a processing unit operatively coupled to each of the plurality of light sensors to receive the sensor signal from each of the plurality of light sensors, to determine a location information of each finger of the user and to issue a command based on the detected location of at least one finger of the user.

17. The method for tracking fingers of a user of claim 16 , wherein the processing unit determines the location information of each finger of the user by identifying a projector pixel corresponding to the sensor signal from a light sensor attached to the corresponding finger of the user.

18. The method for tracking fingers of a user of claim 16 , wherein the issued command causes a sound or a musical note to be synthesized.

19. The method for tracking fingers of a user of claim 16 , wherein the location information of each finger of the user is determined in relation to an image of a piano keyboard comprising a plurality of piano keys.

20. A computer-readable medium embodying a set of instructions implementing a method for tracking fingers of a user, the method comprising:
a. using a projector to project a temporal projector light signal, wherein the temporal projector light signal is encoded, for each pixel of the projector, with an information segment comprising the pixel coordinates of the each pixel of the projector;
b. detecting the temporal projector light signal using a plurality of light sensors, each of the plurality of light sensors being attached to a finger of a user, wherein each of the plurality of light sensors is configured to detect the temporal projector light signal and generate a sensor signal; and
c. using a processing unit operatively coupled to each of the plurality of light sensors to receive the sensor signal from each of the plurality of light sensors, to determine a location information of each finger of the user and to issue a command based on the detected location of at least one finger of the user.

Assignees

• Fuji Xerox Co Ltd

Inventors

• Ma Shang
• Liu Qiong

Classifications

• G10H2230/351

  Spint bell, i.e. mimicking bells, e.g. cow-bells · CPC title

• G10H2220/411

  Light beams · CPC title

• G10H2220/041

  Remote key fingering indicator, i.e. fingering shown on a display separate from the instrument itself or substantially disjoint from the keys · CPC title

• G10H2220/326

  Control glove or other hand or palm-attached control device · CPC title

• G10H3/00

  Instruments in which the tones are generated by electromechanical means · CPC title