Dynamic mixed reality content in virtual reality

Patent metadata

Field	Value
Publication number	US-11676348-B2
Application number	US-202117336776-A
Country	US
Kind code	B2
Filing date	Jun 2, 2021
Priority date	Jun 2, 2021
Publication date	Jun 13, 2023
Grant date	Jun 13, 2023

Abstract

In one embodiment, a method includes using one or more cameras of a mobile computing device to capture one or more images of a first user wearing a VR display device in a real-world environment. The mobile computing device transmits a pose of the mobile computing device with respect to the VR display device to a VR system. The mobile computing device receives from the VR system a VR rendering of a VR environment. The VR rendering is from the perspective of the mobile computing device with respect to the VR display device. The method includes segmenting the first user from the one or more images and generating, in real-time responsive to capturing the one or more images, a MR rendering of the first user in the VR environment. The MR rendering of the first user is based on a compositing of the segmented one or more images of the first user and the VR rendering.

First claim

What is claimed is:

1. A method comprising, by a mobile computing device:
capturing, using one or more cameras of the mobile computing device, one or more images of a first user wearing a virtual reality (VR) display device in a real-world environment, wherein the VR display device is displaying a first VR rendering of a VR environment, and wherein the first VR rendering is from the perspective of the VR display device;
transmitting, from the mobile computing device, a pose of the mobile computing device with respect to the VR display device to a VR system of the VR display device;
receiving, at the mobile computing device and from the VR system, a second VR rendering of the VR environment, wherein the second VR rendering is from the perspective of the mobile computing device with respect to the VR display device;
segmenting the first user from the one or more images; and
generating, at the mobile computing device in real-time responsive to capturing the one or more images, a mixed reality (MR) rendering of the first user in the VR environment, wherein the MR rendering of the first user is based on a compositing of the segmented one or more images of the first user and the second VR rendering.

2. The method of claim 1 , further comprising:
receiving, at the mobile computing device and from the VR system, one or more anchor points in the real-world environment identified from the one or more images.

3. The method of claim 2 , further comprising:
determining a pose of the mobile computing device with respect to the VR display device based on one or more of the anchor points.

4. The method of claim 3 , further comprising:
detecting the mobile computing device has moved;
determining an updated pose of the mobile computing device with respect to the VR display device based on one or more of the anchor points; and
generating, at the mobile computing device in real-time responsive to detecting the mobile computing device has moved, an updated MR rendering of the first user in the VR environment, wherein the updated MR rendering of the first user is based on the updated pose of the mobile computing device with respect to the VR display device.

5. The method of claim 3 , wherein the pose of the mobile computing device with respect to the VR display device is determined based on a triangulation between one or more of the anchor points, the one or more cameras of the mobile computing device, and the VR display device.

6. The method of claim 2 , wherein the one or more anchor points are fixed objects in the real-world environment.

7. The method of claim 1 , further comprising:
toggling, responsive to a selection on the mobile computing device, between:
a VR rendering of the VR environment from the perspective of the first user, and
the MR rendering of the first user in the VR environment from the perspective of the mobile computing device with respect to the VR display device.

8. The method of claim 1 , further comprising:
transmitting, in real-time from the mobile computing device, the generated MR rendering of the first user in the VR environment to an online social network for live display to one or more additional users.

9. The method of claim 1 , wherein segmenting the first user from the one or more images comprises:
identifying, using a segmentation algorithm, a body of the first user within each of the one or more images.

10. The method of claim 1 , further comprising:
receiving, at the mobile computing device and from the VR system, an indication the first user has activated a particular VR application on the VR system; and
presenting, at the mobile computing device responsive to the received indication, a notification that the first user wearing the VR display device has started the particular VR application.

11. The method of claim 1 , wherein the mobile computing device can be used to interact with the VR environment via the MR rendering.

12. The method of claim 11 , wherein interacting with the VR environment comprises one or more of:
modifying a VR object in the VR environment,
creating a VR object in the VR environment,
modifying a MR object in the VR environment,
creating a MR object in the VR environment,
modifying one or more attributes of VR environment,
changing the VR environment from a first VR environment to a second VR environment,
changing a VR application,
recording the MR rendering, or
streaming the MR rendering.

13. The method of claim 11 , wherein interacting with the VR environment comprises one or more of:
creating a VR avatar corresponding to a second user of the mobile computing device, or
interacting with a VR avatar of a third user.

14. One or more computer-readable non-transitory storage media embodying software that is operable when executed by a mobile computing device to:
capture, using one or more cameras of the mobile computing device, one or more images of a first user wearing a virtual reality (VR) display device in a real-world environment, wherein the VR display device is displaying a first VR rendering of a VR environment, and wherein the first VR rendering is from the perspective of the VR display device;
transmit, from the mobile computing device, a pose of the mobile computing device with respect to the VR display device to a VR system of the VR display device;
receive, at the mobile computing device and from the VR system, a second VR rendering of the VR environment, wherein the second VR rendering is from the perspective of the mobile computing device with respect to the VR display device;
segment the first user from the one or more images; and
generate, at the mobile computing device in real-time responsive to capturing the one or more images, a mixed reality (MR) rendering of the first user in the VR environment, wherein the MR rendering of the first user is based on a compositing of the segmented one or more images of the first user and the second VR rendering.

15. The media of claim 14 , wherein the software is further operable when executed to:
receive, at the mobile computing device and from the VR system, one or more anchor points in the real-world environment identified from the one or more images.

16. The method of claim 15 , further comprising:
determine a pose of the mobile computing device with respect to the VR display device based on one or more of the anchor points.

17. The media of claim 16 , further comprising:
detect the mobile computing device has moved;
determine an updated pose of the mobile computing device with respect to the VR display device based on one or more of the anchor points; and
generate, at the mobile computing device in real-time responsive to detecting the mobile computing device has moved, an updated MR rendering of the first user in the VR environment, wherein the updated MR rendering of the first user is based on the updated pose of the mobile computing device with respect to the VR display device.

18. The media of claim 16 , wherein the pose of the mobile computing device with respect to the VR display device is determined based on a triangulation between one or more of the anchor points, the one or more cameras of the mobile computing device, and the VR display device.

19. The media of claim 14 , wherein the one or more anchor points are fixed objects in the real-world environment.

20. A mobile computing device comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable

Assignees

• Meta Platforms Tech Llc

Inventors

• Simpson Sarah Tanner
• Smith Gregory
• Witthuhn Jeffrey
• Huang Ying-Chieh
• Li Shuang
• Zhao Wenliang
• Koch Peter
• Guduru Meghana Reddy
• Pavlidis Ioannis
• Wei Xiang
• Xiao Kevin
• Sheridan Kevin Joseph
• Donselaar Bodhi Keanu
• Camposeco Paulsen Federico Adrian

Classifications

• A63F13/26

  having at least one additional display device, e.g. on the game controller or outside a game booth · CPC title

• A63F13/5255

  according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character · CPC title

• G06T7/11

  Region-based segmentation · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title