Dynamic Mixed Reality Content In Virtual Reality

Patent metadata

Field	Value
Publication number	US-2025166322-A1
Application number	US-202519033911-A
Country	US
Kind code	A1
Filing date	Jan 22, 2025
Priority date	Jun 2, 2021
Publication date	May 22, 2025
Grant date	—

Abstract

In one embodiment, a method includes capturing images of a first user wearing a VR display device in a real-world environment. The method includes receiving a VR rendering of a VR environment. The VR rendering is from the perspective of the mobile computing device with respect to the VR display device. The method includes generating a first MR rendering of the first user in the VR environment., The first MR rendering of the first user is based on a compositing of the images of the first user and the VR rendering. The method includes receiving an indication of a user interaction with one or more elements of the VR environment in the first MR rendering. The method includes generating, in real-time responsive to the indication of the user interaction with the one or more elements, a second MR rendering of the first user in the VR environment. The one or more elements are modified according to the interaction.

First claim

I/We claim:

1 . A method comprising:
capturing one or more images of a user wearing an artificial reality (XR) display device; receiving, from the XR display device, a first rendering of an environment, wherein the first rendering is from the perspective of a capture device with respect to the XR display device; generating, responsive to capturing the one or more images, a second rendering depicting the user in the environment, wherein the second rendering is based on a compositing of the one or more images and the first rendering; receiving an indication of a change in the environment; and generating, responsive to the indication of the change, a third rendering depicting the user in the environment and the change in the environment.

2 . The method of claim 1 , further comprising:
receiving one or more anchor points, in the real-world environment, identified from the one or more images.

3 . The method of claim 2 , further comprising:
determining a pose of the capture device with respect to the XR display device based on one or more of the anchor points.

4 . The method of claim 3 , further comprising:
detecting the capture device has moved; determining an updated pose of the capture device with respect to the XR display device based on one or more of the anchor points; and generating, responsive to detecting the capture device has moved, a fourth rendering of the user in the environment, wherein the fourth rendering of the user is based on the updated pose of the capture device with respect to the XR display device.

5 . The method of claim 3 , wherein the pose of the capture device with respect to the XR display device is determined based on a triangulation between one or more of the anchor points, the one or more cameras of the capture device, and the XR display device.

6 . The method of claim 2 , wherein the one or more anchor points are fixed objects in the real-world environment.

7 . The method of claim 1 , further comprising:
toggling between:
an XR rendering of the environment from the perspective of the user, and
a rendering of the user in the environment from the perspective of the capture device with respect to the XR display device.

8 . The method of claim 1 , further comprising:
transmitting the second or third rendering of the user in the environment to an online social network for live display to one or more additional users.

9 . The method of claim 1 , further comprising:
segmenting a representation of the user from the one or more images; and identifying, using a segmentation algorithm, a body of the user within each of the one or more images, wherein the second or third rendering of the user is based on a compositing of the segmented one or more images of the user and the first rendering.

10 . The method of claim 1 , further comprising:
receiving an indication the user has activated a particular XR application; and presenting, responsive to the received indication, a notification that the user wearing the XR display device has started the particular XR application.

11 . The method of claim 1 , wherein the indication of the change in the environment is received on an interface of the capture device.

12 . The method of claim 1 , wherein the change in the environment comprises one or more of:
a modification of an object in the environment, creation of an object in the environment, a change of the environment from a first environment to a second environment; or creation of a first avatar corresponding to a second user, or an interaction with a second avatar of a third user.

13 . A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to:
capture one or more images of a user wearing an artificial reality (XR) display device; receive, from the XR display device, a first rendering of an environment, wherein the first rendering is from the perspective of a capture device with respect to the XR display device; generate, responsive to capturing the one or more images, a second rendering depicting the user in the environment, wherein the second rendering is based on a compositing of the one or more images and the first rendering; receive an indication of a change in the environment; and generate, responsive to the indication of the change, a third rendering depicting the user in the environment and the change in the environment.

14 . The computer-readable storage medium of claim 13 , wherein the instructions, when executed, further cause the computing system to:
receive one or more anchor points, in the real-world environment, identified from the one or more images.

15 . The computer-readable storage medium of claim 14 , wherein the instructions, when executed, further cause the computing system to:
determine a pose of the capture device with respect to the XR display device based on one or more of the anchor points.

16 . The computer-readable storage medium of claim 15 , wherein the instructions, when executed, further case the computing system to:
detect the capture device has moved; determine an updated pose of the capture device with respect to the XR display device based on one or more of the anchor points; and generate, responsive to detecting the capture device has moved, a fourth rendering of the user in the environment, wherein the fourth rendering of the user is based on the updated pose of the capture device with respect to the XR display device.

17 . The computer-readable storage medium of claim 15 , wherein the pose of the capture device with respect to the XR display device is determined based on a triangulation between one or more of the anchor points, the one or more cameras of the capture device, and the XR display device.

18 . The computer-readable storage medium of claim 13 , wherein the instructions, when executed, further case the computing system to:
transmit the second or third rendering of the user in the environment to an online social network for live display to one or more additional users.

19 . The computer-readable storage medium of claim 13 , wherein the instructions, when executed, further case the computing system to:
segment a representation of the user from the one or more images; and identify, using a segmentation algorithm, a body of the user within each of the one or more images, wherein the second or third rendering of the user is based on a compositing of the segmented one or more images of the user and the first rendering.

20 . A computing system comprising:
one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to:
capture one or more images of a user wearing an artificial reality (XR) display device;
receive, from the XR display device, a first rendering of an environment, wherein the first rendering is from the perspective of a capture device with respect to the XR display device;
generate, responsive to capturing the one or more images, a second rendering depicting the user in the environment, wherein the second rendering is based on a compositing of the one or more images and the first rendering;
receive an indication of a change in the environment; and
generate, responsive to the indication of the change, a third rendering depicting the user in the environment and the change in the environment.

Assignees

• Meta Platforms Tech Llc

Inventors

• Simpson Sarah Tanner
• Smith Gregory
• Witthuhn Jeffrey
• Huang Ying-Chieh
• Li Shuang
• Zhao Wenliang
• Koch Peter
• Guduru Meghana Reddy
• Pavlidis Ioannis
• Wei Xiang
• Xiao Kevin
• Sheridan Kevin Joseph
• Donselaar Bodhi
• Camposeco Paulsen Federico Adrian

Classifications

• G06T2207/30196

  Human being; Person · CPC title

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T7/11

  Region-based segmentation · CPC title