Methods and systems for social sharing head mounted display (HMD) content with a second screen

Patent metadata

Field	Value
Publication number	US-10318230-B2
Application number	US-201715635035-A
Country	US
Kind code	B2
Filing date	Jun 27, 2017
Priority date	Feb 24, 2014
Publication date	Jun 11, 2019
Grant date	Jun 11, 2019

Abstract

Systems and method for processing video frames generated for display on a head mounted display (HMD) to a second screen are provided. In one example, a client system having a processor for generating video frames in response to interactive game play of a video game using the HMD is provided. The generated video frames include in-band regions that include the video game content (VGC) to be displayed on the HMD. A processing device having an input interface for receiving the generated video frames. The processing device including extract logic for extracting a portion of the generated video frames, and a codec unit configured to process the portion of the generated video frames. Further included is a graphics processor unit (GPU) for processing the portion of the generated video frames and formatting for display on the second screen. The portion of the generated video frames processed by the GPU being provided to a first output interface of the processing device. The generated video frames received via the input interface being provided as a pass-through to a second output interface of the processing device. The first output interface of the processing device is connectable to the second screen and the second output interface of the processing device is connectable to the HMD.

First claim

What is claimed is:

1. A system for processing video frames generated for a head mounted display (HMD) for sharing to a second screen, comprising,
a client system having a processor for generating video frames in response to interactive game play of a video game using the HMD, the generated video frames including in-band regions that include video game content (VGC) to be displayed on the HMD and content for out-of-band (OOB) regions; and
a processing device for interfacing between the client system and the HMD, the processing device having an input interface for receiving the generated video frames, the processing device including extract logic for extracting the OOB regions and a graphics processor unit (GPU) for processing a portion of the generated video frames and formatting for display on the second screen, the extracted OOB regions and the portion of the generated video frames being provided to a first output interface of the processing device, the generated video frames received via the input interface being provided as a pass-through to a second output interface of the processing device;
wherein the first output interface of the processing device is connectable to the second screen;
wherein the second output interface of the processing device is connectable to the HMD;
wherein the portion of the generated video frames, selected by the GPU includes video game content (VGC) that is predefined for display to one eye in a display of the HMD, the portion further excluding the OOB regions, and wherein the formatting by the GPU includes performing an un-distortion function to the VGC in the portion, the un-distortion function removes graphic effects that are native for display in the HMD.

2. The system of claim 1 , wherein the processing device includes a codec module for decoding the OOB regions extracted by the extract logic.

3. The system of claim 1 , wherein the formatting by the GPU includes a cropping function and a scaling function, the cropping function and scaling function processed to fit a format of the second screen, that is different than a format of the HMD.

4. The system of claim 1 , wherein the processing device further comprises,
an audio pass-through communicated between the input interface and the first output interface.

5. The system of claim 1 , wherein the processing device further comprises,
a USB module input for connection with the client system, the USB module configured to receive audio and output motion data obtained from the HMD back to the client system.

6. The system of claim 5 , wherein the USB module communicates the received audio from the client system to a 3D audio rendering module for delivery to the HMD, the 3D audio rendering module further configured with an input for receiving audio from the input interface.

7. The system of claim 6 , wherein the processing device further comprises,
a sensor pass-through module for receiving motion data from the HMD, the motion data being communicated to the USB module for communication to the client system, the motion data identifying motion of the HMD that is used by the client system to modify the interactive game play of the video game.

8. The system of claim 1 , further comprising,
a camera input received by the client system, the camera input configured to capture image data of a scene that includes the HMD, the captured image data used by the client system to modify the interactive game play of the video game;
a motion pass-through module in the processing device for receiving motion data from the HMD, the motion data used by the client system to further modify the interactive game play of the video game;
wherein modifications in the interactive game play are rendered in the video frames produced by the client system, the video frames being pass-through the processing device to the HMD and processed by the processing device for sharing to the second screen via the processing device.

9. The system of claim 1 , wherein the extract logic operates on the video frames received from the input interface.

10. A system for processing video frames generated for a head mounted display (HMD) for sharing to a second screen, comprising,
a client system having a processor for generating video frames in response to interactive game play of a video game using the HMD, the generated video frames including in-band regions that include the video game content (VGC) to be displayed on the HMD; and
a processing device having an input interface for receiving the generated video frames, the processing device including,
extract logic for extracting a portion of the generated video frames;
a codec unit configured to process the portion of the generated video frames;
a graphics processor unit (GPU) for processing the portion of the generated video frames and formatting for display on the second screen,
the portion of the generated video frames processed by the GPU being provided to a first output interface of the processing device,
the generated video frames received via the input interface being provided as a pass-through to a second output interface of the processing device;

wherein the first output interface of the processing device is connectable to the second screen;
wherein the second output interface of the processing device is connectable to the HMD;
wherein the formatting by the GPU includes performing an un-distortion function to the VGC in the portion, the un-distortion function removes graphic effects that are for display in the HMD.

11. The system of claim 10 , wherein the codec unit decodes content encoded in H.264 by the client system.

12. The system of claim 10 , wherein the portion of the generated video frames, processed by the GPU includes video game content (VGC) that is predefined for display to one eye in a display of the HMD, the portion excluding out-of-band (OOB) regions.

13. The system of claim 10 , wherein the formatting by the GPU includes a cropping function and a scaling function, the cropping function and scaling function processed to fit a display format of the second screen, that is different than a format of the HMD.

14. The system of claim 10 , wherein a USB input receives audio from the client system and returns output motion data obtained from the HMD back to the client system.

15. The system of claim 14 , wherein the USB input communicates the received audio from the client system to a 3D audio rendering module for delivery to the HMD, the 3D audio rendering module further configured with an input for receiving audio from the input interface.

16. A system for processing video frames generated for a head mounted display (HMD) for sharing to a second screen, comprising,
a client system having a processor for generating video frames in response to interactive game play of a video game using the HMD, the generated video frames including in-band regions that include the video game content (VGC) to be displayed on the HMD; and
a processing device having an input interface for receiving the generated video frames, the processing device including,
extract logic for extracting a portion of the generated video frames;
a codec unit configured to process the portion of the generated video frames;
a graphics processor unit (GPU) for processing the portion of the generated video frames and formatting for display on the second screen,
the portion of the generated video frames processed by the GPU being provided to a first output interface of the processing device,
the generated video frames received via the input interface being provided as a pass-through to a second output interface of the

Assignees

• Sony Interactive Entertainment Inc

Inventors

• Watson Brian
• Mikhailov Anton
• Stafford Jeffrey
• Black Glenn

Classifications

• G02B30/34

  Stereoscopes providing a stereoscopic pair of separated images corresponding to parallactically displaced views of the same object, e.g. three-dimensional [3D] slide viewers · CPC title

• G02B27/0093

  with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking · CPC title

• A63F2300/538

  for performing operations on behalf of the game client, e.g. rendering · CPC title

• A63F13/26 Primary

  having at least one additional display device, e.g. on the game controller or outside a game booth · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title