Method and apparatus for tracking three-dimensional movements of an object using a depth sensing camera

Patent metadata

Field	Value
Publication number	US-10220302-B2
Application number	US-201314059326-A
Country	US
Kind code	B2
Filing date	Oct 21, 2013
Priority date	Jul 27, 2002
Publication date	Mar 5, 2019
Grant date	Mar 5, 2019

Abstract

A controller ( 110 ) for controlling an execution of a game program by a processor for enabling an interactive game to be played by a user includes a body ( 111 ) having a section to be oriented towards a screen when a progress of a game provided via execution of the game apparatus is displayed upon the screen, and at least one photonically detectable (“PD”) element (e.g. 122, 124, 126 , and/or 128 ) assembled with the body, a position of the photonically detectable element within an image being recordable by an image capture device ( 112 ) when the section is oriented at least partly towards the screen, wherein positions of the PD element at different points in time are quantifiable to quantify movement of the body in space.

First claim

What is claimed is:

1. A method for use in obtaining input for an application, comprising:
establishing communications with an image capture device that is capable of providing first output data that includes depth measurements;
using the image capture device to capture images of an object as the object moves in a three-dimensional manner, wherein the object comprises a telephone comprising a hand-held computing and communication device that includes a display screen;
receiving the first output data provided by the image capture device as the image capture device captures images of the object;
receiving additional output data from at least one sensor other than the image capture device that can be used to determine at least one of motion and orientation of the object, wherein the at least one sensor other than the image capture device does not comprise an image capture device;
tracking the three-dimensional movements of the object by mixing the first output data provided by the image capture device with the additional output data provided by the at least one sensor other than the image capture device; and
causing a visual effect to occur on the display screen of the hand-held computing and communication device of the telephone in response to a movement of the object;
wherein the image capture device comprises a three-dimensional camera that measures a depth of a pixel and determines a distance of the pixel from the camera.

2. The method of claim 1 , further comprising:
changing weights that are applied to the first output data and the additional output data in the mixing.

3. The method of claim 1 , wherein the object comprises one or more light sources.

4. The method of claim 3 , wherein the one or more light sources are arranged in an arcuate pattern.

5. The method of claim 3 , wherein the one or more light sources are arranged in a geometric shape.

6. The method of claim 1 , wherein the object comprises a controller.

7. The method of claim 1 , further comprising:
generating input for the application based on the tracked three-dimensional movements of the object.

8. The method of claim 1 , wherein the tracking the three-dimensional movements of the object comprises:
determining position information for the object.

9. The method of claim 1 , wherein the tracking the three-dimensional movements of the object comprises:
determining orientation information for the object.

10. The method of claim 1 , wherein the tracking the three-dimensional movements of the object comprises:
determining acceleration information for the object.

11. The method of claim 1 , wherein the tracking the three-dimensional movements of the object comprises:
determining velocity information for the object.

12. The method of claim 1 , wherein the tracking the three-dimensional movements of the object comprises:
determining at least one of a tilt, pitch, yaw, and roll for the object.

13. A computer program product comprising a non-transitory medium for embodying a computer program for input to a computer and a computer program embodied in the non-transitory medium for causing the computer to perform steps comprising:
establishing communications with an image capture device that is capable of providing first output data that includes depth measurements;
using the image capture device to capture images of an object as the object moves in a three-dimensional manner, wherein the object comprises a telephone comprising a hand-held computing and communication device that includes a display screen;
receiving the first output data provided by the image capture device as the image capture device captures images of the object;
receiving additional output data from at least one sensor other than the image capture device that can be used to determine at least one of motion and orientation of the object, wherein the at least one sensor other than the image capture device does not comprise an image capture device;
tracking the three-dimensional movements of the object by mixing the first output data provided by the image capture device with the additional output data provided by the at least one sensor other than the image capture device; and
causing a visual effect to occur on the display screen of the hand-held computing and communication device of the telephone in response to a movement of the object;
wherein the image capture device comprises a three-dimensional camera that measures a depth of a pixel and determines a distance of the pixel from the camera.

14. The computer program product of claim 13 , wherein the computer program further causes the computer to perform a step comprising:
changing weights that are applied to the first output data and the additional output data in the mixing.

15. The computer program product of claim 13 , wherein the object comprises one or more light sources.

16. The computer program product of claim 15 , wherein the one or more light sources are arranged in an arcuate pattern.

17. The computer program product of claim 15 , wherein the one or more light sources are arranged in a geometric shape.

18. The computer program product of claim 13 , wherein the object comprises a controller.

19. The computer program product of claim 13 , wherein the computer program further causes the computer to perform a step comprising:
generating input for an application based on the tracked three-dimensional movements of the object.

20. The computer program product of claim 13 , wherein the tracking the three-dimensional movements of the object comprises:
determining position information for the object.

21. The computer program product of claim 13 , wherein the tracking the three-dimensional movements of the object comprises:
determining orientation information for the object.

22. The computer program product of claim 13 , wherein the tracking the three-dimensional movements of the object comprises:
determining acceleration information for the object.

23. The computer program product of claim 13 , wherein the tracking the three-dimensional movements of the object comprises:
determining velocity information for the object.

24. The computer program product of claim 13 , wherein the tracking the three-dimensional movements of the object comprises:
determining at least one of a tilt, pitch, yaw, and roll for the object.

25. An apparatus, comprising:
an image capture device that is capable of providing first output data that includes depth measurements; and
a processing system that is communicatively coupled to the image capture device, wherein the processing system is configured to execute steps comprising using the image capture device to capture images of an object as the object moves in a three-dimensional manner, receiving the first output data provided by the image capture device as the image capture device captures images of the object, receiving additional output data from at least one sensor other than the image capture device that can be used to determine at least one of motion and orientation of the object, and tracking the three-dimensional movements of the object by mixing the first output data provided by the image capture device with the additional output data provided by the at least one sensor other than the image capture device;
wherein the object comprises a telephone comprising a hand-held computing and communication device that includes a display screen;
wherein the processing

Assignees

• Sony Computer Entertainment Inc
• Sony Interactive Entertainment Inc

Inventors

• Zalewski Gary M
• Marks Richard
• Mao Xiaodong

Classifications

• G06T2207/30204

  Marker · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• A63F2300/1093

  using visible light · CPC title