Handheld device for spectator viewing of an interactive application

Patent metadata

Field	Value
Publication number	US-9084938-B2
Application number	US-201414503218-A
Country	US
Kind code	B2
Filing date	Sep 30, 2014
Priority date	Oct 12, 2010
Publication date	Jul 21, 2015
Grant date	Jul 21, 2015

Abstract

A handheld device is provided, comprising: a sensor configured to generate sensor data for determining and tracking a position and orientation of the handheld device during an interactive session of an interactive application presented on a main display, the interactive session being defined for interactivity between a user and the interactive application; a communications module configured to send the sensor data to a computing device, the communications module being further configured to receive from the computing device a spectator video stream of the interactive session that is generated based on a state of the interactive application and the tracked position and orientation of the handheld device, the state of the interactive application being determined based on the interactivity between the user and the interactive application; a display configured to render the spectator video stream.

First claim

What is claimed is:

1. A handheld device, comprising:
a sensor configured to generate sensor data for determining and tracking a position and orientation of the handheld device during an interactive session of an interactive application presented on a main display, the interactive session being defined for interactivity between a user and the interactive application;
a communications module configured to send the sensor data to a computing device, the communications module being further configured to receive from the computing device a spectator video stream of the interactive session that is generated based on a state of the interactive application and the tracked position and orientation of the handheld device, the state of the interactive application being determined based on the interactivity between the user and the interactive application;
a display configured to render the spectator video stream;
wherein the position and orientation of the handheld device define a position and orientation of a virtual viewpoint within a virtual space defined by the interactive application; and
wherein the spectator video stream includes images of the virtual space captured from the perspective of the virtual viewpoint.

2. The handheld device of claim 1 , wherein the state of the interactive application is determined from interactive input, the interactive input being received from a controller device during the interactive session.

3. The handheld device of claim 1 ,
wherein the interactive session defines interactivity between the user and an application video stream rendered on the main display, the application video stream being generated based on the state of the interactive application.

4. The handheld device of claim 1 ,
wherein the sensor is defined by one or more of an accelerometer, a magnetometer, and/or a gyroscope; and
wherein the sensor data includes inertial data generated by the sensor.

5. The handheld device of claim 1 ,
wherein the sensor is defined by an image capture device; and
wherein the sensor data is defined by captured image data of an interactive environment in which the interactivity occurs.

6. The handheld device of claim 5 , wherein determining and tracking the position and orientation of the handheld device is based on identification of one or more markers in the interactive environment.

7. A handheld device, comprising:
a sensor configured to generate sensor data for determining and tracking a position and orientation of the handheld device during an interactive session of an interactive application presented on a main display, the interactive session being defined for interactivity between a user and the interactive application;
a communications module configured to send the sensor data to a computing device, the communications module being further configured to receive application state data from the computing device that is generated based on a state of the interactive application, the state of the interactive application being determined based on the interactivity between the user and the interactive application;
a spectator video stream generator configured to generate a spectator video stream based on the application state data and the tracked position and orientation of the handheld device;
a display configured to render the spectator video stream.

8. The handheld device of claim 7 , further comprising:
an image capture device configured to capture an environmental video stream of an interactive environment in which the interactivity between the user and the interactive application occurs;
wherein the spectator video stream generator is configured to generate the spectator video stream based on the environmental video stream.

9. The handheld device of claim 8 , wherein the spectator video stream generator is configured to generate the spectator video stream by augmenting the environmental video stream with a virtual element.

10. The handheld device of claim 8 , wherein the spectator video stream generator is configured to generate the spectator video stream by replacing at least a portion of the user that is detected in the environmental video stream with a virtual element.

11. The handheld device of claim 10 , wherein the replacing at least a portion of the user includes replacing the detected user in the environmental video stream with a character of the interactive application controlled by the user.

12. The handheld device of claim 7 , wherein the spectator video stream is generated based on a tracked position and orientation of the handheld device relative to a tracked position of the user.

13. The handheld device of claim 7 ,
wherein the sensor is defined by one or more of an accelerometer, a magnetometer, and/or a gyroscope; and
wherein the sensor data includes inertial data generated by the sensor.

14. A handheld device, comprising:
a sensor configured to generate sensor data for determining and tracking a position and orientation of the handheld device during an interactive session of an interactive application presented on a main display, the interactive session being defined for interactivity between a user and the interactive application;
a communications module configured to send the sensor data to a computing device, the communications module being further configured to receive from the computing device a spectator video stream of the interactive session that is generated based on a state of the interactive application and the tracked position and orientation of the handheld device, the state of the interactive application being determined based on the interactivity between the user and the interactive application;
a display configured to render the spectator video stream;
wherein the spectator video stream is generated based on the position and orientation of the handheld device relative to a tracked position of the user during the interactive session;
wherein the position and orientation of the handheld device relative to the position of the user define a position and orientation of a virtual viewpoint within a virtual space defined by the interactive application; and
wherein an object within a virtual environment of the interactive application is mapped to the position of the user;
wherein the spectator video stream includes images of the virtual space captured from the perspective of the virtual viewpoint, the object being included in the images of the spectator video stream when the handheld device is oriented towards the position of the user.

15. The handheld device of claim 14 , wherein the object is controlled by the user.

16. The handheld device of claim 15 , wherein the object is a character of the interactive application.

17. The handheld device of claim 14 , wherein the position of the user is determined based on the position of a controller device operated by the user.

18. The handheld device of claim 17 , further comprising,
an image capture device configured to capture an environmental video stream of an interactive environment in which the interactivity between the user and the interactive application occurs;
wherein the position of the user and the position and orientation of the handheld device relative to the position of the user are determined at least in part based on the environmental video stream.

Assignees

• Sony Computer Entertainment Inc

Inventors

• Osman Steven
• Zhu Yunpeng
• Stafford Jeffrey Roger

Classifications

• A63F13/428

  involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes · CPC title

• A63F13/211 Primary

  using inertial sensors, e.g. accelerometers or gyroscopes · CPC title

• A63F13/235

  using a wireless connection, e.g. infrared or piconet · CPC title

• A63F13/86

  Watching games played by other players · CPC title

• A63F2300/403

  Connection between platform and handheld device · CPC title