Dynamic mixed reality content in virtual reality

Patent metadata

Field	Value
Publication number	US-12272012-B2
Application number	US-202318321712-A
Country	US
Kind code	B2
Filing date	May 22, 2023
Priority date	Jun 2, 2021
Publication date	Apr 8, 2025
Grant date	Apr 8, 2025

Abstract

In one embodiment, a method includes capturing images of a first user wearing a VR display device in a real-world environment. The method includes receiving a VR rendering of a VR environment. The VR rendering is from the perspective of the mobile computing device with respect to the VR display device. The method includes generating a first MR rendering of the first user in the VR environment. The first MR rendering of the first user is based on a compositing of the images of the first user and the VR rendering. The method includes receiving an indication of a user interaction with one or more elements of the VR environment in the first MR rendering. The method includes generating, in real-time responsive to the indication of the user interaction with the one or more elements, a second MR rendering of the first user in the VR environment. The one or more elements are modified according to the interaction.

First claim

What is claimed is:

1. A method comprising, by a mobile computing device:
capturing one or more images of a first user wearing a virtual reality (VR) display device in a real-world environment;
receiving, from a VR system of the VR display device, a VR rendering of a VR environment, wherein the VR rendering is from the perspective of the mobile computing device with respect to the VR display device;
generating, in real-time responsive to capturing the one or more images, a first mixed reality (MR) rendering of the first user in the VR environment, wherein the first MR rendering of the first user is based on a compositing of the one or more images of the first user and the VR rendering;
receiving, by the mobile computing device, an indication of a user interaction with one or more elements of the VR environment in the first MR rendering; and
generating, in real-time responsive to the indication of the user interaction with the one or more elements, a second MR rendering of the first user in the VR environment, wherein the one or more elements have been modified according to the user interaction.

2. The method of claim 1 , further comprising:
receiving, at the mobile computing device and from the VR system, one or more anchor points in the real-world environment identified from the one or more images.

3. The method of claim 2 , further comprising:
determining a pose of the mobile computing device with respect to the VR display device based on at least one of the one or more anchor points.

4. The method of claim 3 , further comprising:
detecting the mobile computing device has moved;
determining an updated pose of the mobile computing device with respect to the VR display device based on at least one of the one or more anchor points; and
generating, at the mobile computing device in real-time responsive to detecting the mobile computing device has moved, a third MR rendering of the first user in the VR environment, wherein the third MR rendering of the first user is based on the updated pose of the mobile computing device with respect to the VR display device.

5. The method of claim 3 , wherein the pose of the mobile computing device with respect to the VR display device is determined based on a triangulation between at least one of the one or more anchor points, one or more cameras of the mobile computing device, and the VR display device.

6. The method of claim 2 , wherein the one or more anchor points are fixed objects in the real-world environment.

7. The method of claim 1 , further comprising:
toggling, responsive to a selection on the mobile computing device, between:
a VR rendering of the VR environment from the perspective of the first user, and
the first or second MR rendering of the first user in the VR environment from the perspective of the mobile computing device with respect to the VR display device.

8. The method of claim 1 , further comprising:
transmitting, in real-time from the mobile computing device, the first or second MR rendering of the first user in the VR environment to an online social network for live display to one or more additional users.

9. The method of claim 1 , further comprising:
segmenting the first user from the one or more images; and
identifying, using a segmentation algorithm, a body of the first user within each of the one or more images, wherein the first or second MR rendering of the first user is based on a compositing of the segmented one or more images of the first user and the VR rendering.

10. The method of claim 1 , further comprising:
receiving, at the mobile computing device and from the VR system, an indication the first user has activated a particular VR application on the VR system; and
presenting, at the mobile computing device responsive to the received indication the first user has activated the particular VR application, a notification that the first user wearing the VR display device has started the particular VR application.

11. The method of claim 1 , wherein the indication of the user interaction with the one or more elements of the VR environment is received on an interface of the mobile computing device.

12. The method of claim 1 , wherein the user interaction with the one or more elements of the VR environment comprises one or more of:
modifying a VR object in the VR environment,
creating a VR object in the VR environment,
modifying a MR object in the VR environment,
creating a MR object in the VR environment,
modifying one or more attributes of VR environment,
changing the VR environment from a first VR environment to a second VR environment,
changing a VR application,
recording the first or second MR rendering, or
streaming the first or second MR rendering.

13. The method of claim 1 , wherein the user interaction with the one or more elements of the VR environment comprises one or more of:
creating a VR avatar corresponding to a second user of the mobile computing device, or
interacting with a VR avatar of a third user.

14. One or more computer-readable non-transitory storage media embodying software that is operable when executed by a mobile computing device to:
capture one or more images of a first user wearing a virtual reality (VR) display device in a real-world environment;
receive, from a VR system of the VR display device, a VR rendering of a VR environment, wherein the VR rendering is from the perspective of the mobile computing device with respect to the VR display device;
generate, in real-time responsive to capturing the one or more images, a first mixed reality (MR) rendering of the first user in the VR environment, wherein the first MR rendering of the first user is based on a compositing of the one or more images of the first user and the VR rendering;
receive, by the mobile computing device, an indication of a user interaction with one or more elements of the VR environment in the first MR rendering; and
generate, in real-time responsive to the indication of the user interaction with the one or more elements, a second MR rendering of the first user in the VR environment, wherein the one or more elements have been modified according to the user interaction.

15. The one or more computer-readable non-transitory storage media of claim 14 , wherein the software is further operable when executed to:
receive, at the mobile computing device and from the VR system, one or more anchor points in the real-world environment identified from the one or more images.

16. The one or more computer-readable non-transitory storage media of claim 15 , further comprising:
determine a pose of the mobile computing device with respect to the VR display device based on at least one of the one or more anchor points.

17. The one or more computer-readable non-transitory storage media of claim 16 , further comprising:
detect the mobile computing device has moved;
determine an updated pose of the mobile computing device with respect to the VR display device based on at least one of the one or more anchor points; and
generate, at the mobile computing device in real-time responsive to detecting the mobile computing device has moved, a third MR rendering of the first user in the VR environment, wherein the third MR rendering of the first user is based on the updated pose of the mobile computing device with respect to the VR display device.

18. The one or more computer-readable non-transitory storage media of claim 16 , wherein the pose of the mobile computing device with respect to the VR display device is determined based on a triangulation between at

Assignees

• Meta Platforms Tech Llc

Inventors

• Simpson Sarah Tanner
• Smith Gregory
• Witthuhn Jeffrey
• Huang Ying-Chieh
• Li Shuang
• Zhao Wenliang
• Koch Peter
• Guduru Meghana Reddy
• Pavlidis Ioannis
• Wei Xiang
• Xiao Kevin
• Sheridan Kevin Joseph
• Donselaar Bodhi Keanu
• Camposeco Paulsen Federico Adrian

Classifications

• G06T2207/30196

  Human being; Person · CPC title

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T7/11

  Region-based segmentation · CPC title