Systems and methods for providing feedback to a user while interacting with content

Patent metadata

Field	Value
Publication number	US-9746984-B2
Application number	US-201414503275-A
Country	US
Kind code	B2
Filing date	Sep 30, 2014
Priority date	Aug 19, 2014
Publication date	Aug 29, 2017
Grant date	Aug 29, 2017

Abstract

Systems and methods for providing feedback to a user are described. One of the methods includes sending interactive data for rendering on a head mounted display (HMD). The HMD is configured for attachment to a head of a user for playing a game. The method further includes receiving image data associated with a body part of the user while the HMD is worn by the user and the HMD renders interactive content. The method includes tracking a spatial position associated with the body part of the user from the image data of the user. In the method, physical placement of a remote-controlled device at the spatial position associated with the body part of the user is rendered in the HMD as virtual contact with an object in the interactive content.

First claim

What is claimed is:

1. A method comprising:
sending interactive data for rendering on a head mounted display (HMD), the HMD configured for attachment to a head of a user for playing a game;
receiving image data associated with a body part of the user while the HMD is worn by the user and the HMD renders interactive content;
tracking a spatial position associated with the body part of the user from the image data associated with the body part of the user; and
for a state of rendering by the HMD and for the spatial position associated with the body part, controlling a position of a remotely-controlled device (RCD) to fly and place the RCD proximate to the spatial position associated with the body part, wherein the placement of the RCD at the spatial position associated with the body part of the user is rendered in the HMD as virtual contact with an object in the interactive content.

2. The method of claim 1 , further comprising:
determining a spatial position of the object displayed within the interactive content;
determining a spatial position of a representation of the body part, the representation of the body part displayed within the interactive content;
determining whether the spatial position of the representation of the body part is within a pre-determined range from the spatial position of the object,
wherein controlling the position of the RCD is performed when the spatial position of the representation of the body part is within the pre-determined range from the spatial position of the object.

3. The method of claim 1 , further comprising scaling a position of the body part that is represented within the image data to the spatial position associated with the body part within a real-world in which the user is located.

4. The method of claim 1 , wherein the image data associated with the body part includes image data received from an image capturing device that captures an image of the body part or an image of a position device that is attached to the body part.

5. The method of claim 1 , wherein the body part includes a hand or a leg or the head of the user.

6. The method of claim 1 , wherein receiving the image data associated with the body part of the user while the HMD is worn by the user and the HMD renders the interactive content comprises receiving the image data from an image capturing device at a time the HMD is displaying the interactive content and the HMD is worn by the user.

7. The method of claim 1 , wherein tracking the spatial position associated with the body part from the image data includes determining x, y, and z co-ordinates of the body part or of a position device attached to the body part from a reference point in a real-world environment.

8. The method of claim 1 , wherein the interactive data includes image data used to render a virtual game.

9. The method of claim 1 , wherein the state of rendering by the HMD is based on a state of an interactive program that is executed to generate the interactive data that is further rendered to display the interactive content.

10. The method of claim 1 , wherein the interactive content includes a virtual reality scene or an augmented reality scene or a video or streaming data.

11. The method of claim 1 , wherein the RCD includes a drone or a robot or an exercise machine.

12. The method of claim 1 , wherein the RCD is proximate to the spatial position associated with the body part when a contact surface of the RCD touches the body part or a position device that is attached to the body part.

13. The method of claim 1 , wherein the RCD is proximate to the spatial position associated with the body part when a contact surface of the RCD has the same position as that of the body part or of a position device that is attached to the body part.

14. The method of claim 1 , wherein the RCD is proximate to the spatial position associated with the body part when a contact surface of the RCD is within a pre-determined distance of the body part or of a position device that is attached to the body part.

15. A system comprising:
a head mounted display (HMD) for displaying a game, the head mounted display for mounting on a head of a user for playing the game; and
a game console coupled to the head mounted display, the game console including a gaming processor for:
sending interactive content for rendering in the HMD;
receiving image data associated with a body part of the user while the HMD is worn by the user and the HMD renders the interactive content;
tracking a spatial position associated with the body part of the user from the image data;
generating an instruction based on a current state of rendering by the HMD and the spatial position associated with the body part; and
sending the instruction to a remotely-controlled device (RCD) to control a position of the RCD to fly and place the RCD proximate to the spatial position associated with the body part, wherein the placement of the RCD at the spatial position associated with body part of the user is rendered in the HMD as virtual contact with an object in the interactive content.

16. The system of claim 15 , wherein the gaming processor is for:
determining a spatial position of the object displayed within the interactive content;
determining a spatial position of a representation of the body part, the representation displayed within the interactive content; and
determining whether the spatial position of the representation of the body part is within a pre-determined range from the spatial position of the object,
wherein sending the instruction to control the position of the RCD is performed when the spatial position of the representation of the body part is within the pre-determined range from the spatial position of the object.

17. The system of claim 15 , wherein the gaming processor is for:
determining a spatial position of a representation of the body part from the image data;
associating the spatial position of the representation of the body part in the image data with a spatial position of a representation of the body part in the interactive content;
determining a spatial position of a virtual object in the interactive content;
determining whether the spatial position of the representation of the body part in the interactive content is within a pre-determined distance from the spatial position of the virtual object in the interactive content,
wherein sending the instruction to control the position of the RCD is performed when the spatial position of the representation of the body part in the interactive content is within the pre-determined distance from the spatial position of the virtual object in the interactive content.

18. The system of claim 15 , wherein the RCD includes a drone.

19. A remotely-controlled device (RCD) comprising:
a communication circuit for receiving an instruction from a game console;
a mechanical component;
a contact surface; and
a position controller coupled to the mechanical component and to the communication circuit, the position controller for controlling a position of the contact surface based on the instruction, the position controller for controlling the mechanical component to fly to further control the position of the contact surface, the position controller to control the position of the contact surface to place the contact surface proximate to a spatial position associated with a body part of a user during a play of a game, the game having interactive content, wherein the placement of the contact surface at the spatial position associated with the body part of t

Assignees

• Sony Interactive Entertainment Inc

Inventors

• Stafford Jeffrey Roger

Classifications

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• A63F2300/1087

  comprising photodetecting means, e.g. a camera · CPC title

• G06F3/048 Primary

  Interaction techniques based on graphical user interfaces [GUI] · CPC title

• H04N13/044

  Electricity · mapped topic

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title