Interfaces and techniques to retarget 2D screencast videos into 3D tutorials in virtual reality

Patent metadata

Field	Value
Publication number	US-11030796-B2
Application number	US-201816163428-A
Country	US
Kind code	B2
Filing date	Oct 17, 2018
Priority date	Oct 17, 2018
Publication date	Jun 8, 2021
Grant date	Jun 8, 2021

Abstract

Embodiments of the present invention provide systems, methods, and computer storage media which retarget 2D screencast video tutorials into an active VR host application. VR-embedded widgets can render on top of a VR host application environment while the VR host application is active. Thus, VR-embedded widgets can provide various interactive tutorial interfaces directly inside the environment of the VR host application. For example, VR-embedded widgets can present external video content, related information, and corresponding interfaces directly in a VR painting environment, so a user can simultaneously access external video (e.g., screencast video tutorials) and a VR painting. Possible VR-embedded widgets include a VR-embedded video player overlay widget, a perspective thumbnail overlay widget (e.g., a user-view thumbnail overlay, an instructor-view thumbnail overlay, etc.), an awareness overlay widget, a tutorial steps overlay widget, and/or a controller overlay widget, among others.

First claim

What is claimed is:

1. A method comprising:
intercepting, by a virtual reality (VR)-embedded video application, a rendered three-dimensional (3D) environment transmitted by a VR design application to a VR display before the VR display receives the rendered 3D environment;
rendering, by the VR-embedded video application, a composite 3D environment by rendering a VR-embedded widget on top of the rendered 3D environment;
outputting, by the VR-embedded video application, the composite 3D environment to the VR display;
evaluating, by the VR-embedded video application, VR inputs transmitted to the VR design application before the VR design application receives the VR inputs;
intercepting, by the VR-embedded video application, a first set of the VR inputs that interact with the VR-embedded widget in the composite 3D environment; and
determining, by the VR-embedded video application, not to intercept a second set of the VR inputs that do not interact with the VR-embedded widget in the composite 3D environment,
wherein the VR-embedded widget is configured to present at least one of an external two-dimensional (2D) screencast video or a three-dimensional (3D) simulation scene associated with the external 2D screencast video.

2. The method of claim 1 , wherein the VR-embedded widget is configured to render the external 2D screencast video stereoscopically.

3. The method of claim 1 , the method further comprising rendering, based on a determination that the VR-embedded widget is not visible in a field of view, a view-fixed awareness overlay widget configured to present visual information about the external 2D screencast video.

4. The method of claim 3 , wherein the view-fixed awareness overlay widget includes a video timeline with icons representing events of the external 2D screencast video, wherein the icons are configured to temporarily increase size as corresponding events approach.

5. The method of claim 1 , the method further comprising automatically detecting events in the external 2D screencast video by analyzing at least one of the external 2D screencast video or corresponding log data.

6. The method of claim 5 , wherein the VR-embedded widget includes a video timeline with icons representing the detected events.

7. The method of claim 1 , wherein the external 2D screencast video comprises a live stream of an instructor retargeted in real-time, and wherein the VR-embedded widget is configured to act as a communication bridge.

8. A system comprising:
one or more hardware processors and memory configured to provide computer program instructions to the one or more hardware processors; and
a means for rendering a VR-embedded widget on top of a rendered three-dimensional (3D) environment of an active VR host application by using the one or more hardware processors to:
evaluate VR inputs transmitted to the active VR host application before the active VR host application receives the VR inputs;
intercept a first set of the VR inputs that interact with the VR-embedded widget in the composite 3D environment; and
determine not to intercept a second set of the VR inputs that do not interact with the VR-embedded widget in the composite 3D environment,

wherein the VR-embedded widget is configured to present at least one of an external two-dimensional (2D) screencast video or a three-dimensional (3D) simulation scene associated with the external 2D screencast video.

9. The system of claim 8 , wherein the VR host application comprises a rendering and event system, and wherein the system additionally comprises an injection component configured to inject the rendering and event system of the VR host application into code for the VR-embedded widget.

10. The system of claim 8 , additionally comprising an event detector configured to automatically detect events in the external 2D screencast video by analyzing at least one of the external 2D screencast video or corresponding log data.

11. The system of claim 10 , wherein the VR-embedded widget is configured to represent the detected events as icons on a video timeline.

12. A system comprising:
one or more hardware processors and memory configured to provide computer program instructions to the one or more hardware processors;
an injection component, of a virtual reality (VR)-embedded video application, configured to use the one or more hardware processors to:
intercept a rendered three-dimensional (3D) environment transmitted by a VR design application to a VR display before the VR display receives the rendered 3D environment;
render a composite 3D environment by rendering a VR-embedded widget on top of the rendered 3D environment;
evaluate VR inputs transmitted to the VR design application before the VR design application receives the VR inputs;
intercept a first set of the VR inputs that interact with the VR-embedded widget in the composite 3D environment and pass the first set of VR inputs to the VR-embedded widget;
determine not to intercept a second set of the VR inputs into the VR design application that do not interact with the VR-embedded widget in the composite 3D environment; and
output the composite 3D environment to the VR display,

wherein the VR-embedded widget is configured to present at least one of an external two-dimensional (2D) screencast video or a three-dimensional (3D) simulation scene associated with the external 2D screencast video.

13. The system of claim 12 , wherein the VR-embedded widget is configured to render the external 2D screencast video stereoscopically.

14. The system of claim 12 , the VR-embedded video application further configured to render, based on a determination that the VR-embedded widget is not visible in a field of view, a view-fixed awareness overlay widget configured to present visual information about the external 2D screencast video.

15. The system of claim 14 , wherein the view-fixed awareness overlay widget includes a video timeline with icons representing events of the external 2D screencast video, wherein the icons are configured to temporarily increase size as corresponding events approach.

16. The system of claim 12 , the VR-embedded video application further configured to automatically detect events in the external 2D screencast video by analyzing at least one of the external 2D screencast video or corresponding log data.

17. The system of claim 16 , wherein the VR-embedded widget includes a video timeline with icons representing the detected events.

18. The system of claim 12 , wherein the external 2D screencast video comprises a live stream of an instructor retargeted in real-time, and wherein the VR-embedded widget is configured to act as a communication bridge.

Assignees

• Adobe Inc

Inventors

• Nguyen Cuong
• Diverdi Stephen Joseph
• Kumaravel Balasaravanan Thoravi

Classifications

• H04N21/443

  OS processes, e.g. booting an STB, implementing a Java virtual machine in an STB or power management in an STB · CPC title

• H04N21/8547

  involving timestamps for synchronizing content · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T19/003

  Navigation within 3D models or images · CPC title