Providing shared augmented reality environments within video calls

Patent metadata

Field	Value
Publication number	US-2023300292-A1
Application number	US-202217818596-A
Country	US
Kind code	A1
Filing date	Aug 9, 2022
Priority date	Mar 15, 2022
Publication date	Sep 21, 2023
Grant date	—

Abstract

Systems, methods, client devices, and non-transitory computer-readable media are disclosed for enabling video calls which facilitate shared augmented reality (AR) background environments. In particular, the disclosed systems can establish a video call between client devices. In addition, the disclosed systems can enable client devices to segment one or more participants captured via videos on the client devices from captured backgrounds. Moreover, the disclosed systems can enable the client devices to render, in place of the segmented backgrounds, AR background environments to place captured videos of the one or more participants within a shared AR background space to create the perception that the participants of the video call are present in the same location (or setting). Furthermore, the disclosed systems can enable the client devices to render the AR environments as shared 360 AR environments that render synchronized 360 AR spaces and/or AR effects across the client devices participating in the video call (e.g., utilizing an AR data channel).

First claim

What is claimed is:

1 . A computer-implemented method comprising:
conducting, by a client device, a video call with a participant device by receiving video data through a video data channel established for the video call from the participant device; rendering, within a digital video call interface, a first video captured by the client device within a first portion of a three-dimensional (3D) shared augmented reality (AR) space; and displaying, within the digital video call interface, a second video from the participant device by rendering video data received through the video data channel, the second video portraying a second portion of the 3D shared AR space.

2 . The computer-implemented method of claim 1 , further comprising receiving AR data corresponding to the 3D shared AR space through an AR data channel from the participant device to render the first video within the first portion of the 3D shared AR space.

3 . The computer-implemented method of claim 1 , wherein rendering the first video within the first portion of the 3D shared AR space comprises:
generating an AR background texture for the 3D shared AR space; and replacing a background of the first video with a portion of the AR background texture corresponding to the first portion of the 3D shared AR space.

4 . The computer-implemented method of claim 1 , further comprising:
segmenting a user depicted within the first video from a background of the first video to generate a segmented user portion; and inserting the segmented user portion of the first video into the first portion of the 3D shared AR space.

5 . The computer-implemented method of claim 1 , further comprising transmitting, through an AR data channel, AR data corresponding to the 3D shared AR space to the participant device to cause the participant device to render the second video within the second portion of the 3D shared AR space.

6 . The computer-implemented method of claim 1 , further comprising:
detecting, by the client device, user movement during the video call; and rendering, within the digital video call interface, the first video captured by the client device within a third portion of the 3D shared AR space based on the detected user movement.

7 . The computer-implemented method of claim 6 , wherein the user movement comprises detected movement of a participant depicted within the first video or client device movement of the client device.

8 . The computer-implemented method of claim 1 , further comprising:
receiving, by the client device, a user interaction with the 3D shared AR space; modifying the 3D shared AR space corresponding to the first video based on the user interaction by adding an AR element to the 3D shared AR space or modifying an existing AR element within the 3D shared AR space; and transmitting, through an AR data channel, AR data corresponding to the user interaction to the participant device to cause the participant device to modify the 3D shared AR space corresponding to the second video.

9 . The computer-implemented method of claim 8 , wherein a modification to the 3D shared AR space corresponding to the first video and corresponding to the second video is positioned at a matching portion of the 3D shared AR space.

10 . The computer-implemented method of claim 8 , further comprising saving a modification to the 3D shared AR space for subsequent video calls between the client device and the participant device.

11 . A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to:
conduct, by a client device, a video call with a participant device by receiving video data through a video data channel established for the video call from the participant device; render, within a digital video call interface, a first video captured by the client device within a first portion of a three-dimensional (3D) shared augmented reality (AR) space; and display, within the digital video call interface, a second video from the participant device by rendering video data received through the video data channel, the second video portraying a second portion of the 3D shared AR space.

12 . The non-transitory computer-readable medium of claim 11 , wherein rendering the first video within the first portion of the 3D shared AR space comprises:
generating an AR background texture for the 3D shared AR space; and replacing a background of the first video with a portion of the AR background texture corresponding to the first portion of the 3D shared AR space.

13 . The non-transitory computer-readable medium of claim 11 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to:
segment a user depicted within the first video from a background of the first video to generate a segmented user portion; and insert the segmented user portion of the first video into the first portion of the 3D shared AR space.

14 . The non-transitory computer-readable medium of claim 11 , wherein the first portion and the second portion of the 3D shared AR space comprise different views of a synchronized AR environment.

15 . The non-transitory computer-readable medium of claim 11 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to:
receive, by the client device, a selection of an additional 3D shared AR space; render, within the digital video call interface, the first video captured by the client device within a first portion of the additional 3D shared AR space; and transmit, through an AR data channel, AR data corresponding to the additional 3D shared AR space to the participant device to cause the participant device to render the second video within a second portion of the additional 3D shared AR space.

16 . A system comprising:
at least one processor; and at least one non-transitory computer-readable medium comprising instructions that, when executed by the at least one processor, cause the system to:
conduct, by a client device, a video call with a participant device by receiving video data through a video data channel established for the video call from the participant device;
render, within a digital video call interface, a first video captured by the client device within a first portion of a three-dimensional (3D) shared augmented reality (AR) space; and
display, within the digital video call interface, a second video from the participant device by rendering video data received through the video data channel, the second video portraying a second portion of the 3D shared AR space.

17 . The system of claim 16 , further comprising instructions that, when executed by the at least one processor, cause the system to receive AR data corresponding to the 3D shared AR space through an AR data channel from the participant device to render the first video within the first portion of the 3D shared AR space.

18 . The system of claim 16 , wherein rendering the first video within the first portion of the 3D shared AR space comprises:
generating an AR background texture for the 3D shared AR space; and replacing a background of the first video with a portion of the AR background texture corresponding to the first portion of the 3D shared AR space.

19 . The system of claim 16 , further comprising instructions that, when executed by the at least one processor, cause the system to:
detect, by the client device, user movement during the video call, the user moveme

Assignees

• Meta Platforms Inc

Inventors

• Barbosa Da Silva Marcus Vinicius
• Cohen David Marc
• Dzieza Emmett
• Del Rio Lazo Dalia Antoinette

Classifications

• G06Q10/40

  Business processes related to social networking or social networking services · CPC title

• H04N7/157 Primary

  defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827) · CPC title

• H04N7/141 Primary

  between two video terminals, e.g. videophone (telephonic communication systems combined with television receiver for reception of entertainment or information matter H04M11/085) · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title