Spectator view of virtual and physical objects

Patent metadata

Field	Value
Publication number	US-12379602-B2
Application number	US-202418735799-A
Country	US
Kind code	B2
Filing date	Jun 6, 2024
Priority date	Mar 10, 2020
Publication date	Aug 5, 2025
Grant date	Aug 5, 2025

Abstract

A user may interact and view virtual elements such as avatars and objects and/or real world elements in three-dimensional space in an augmented reality (AR) session. The system may allow one or more spectators to view from a stationary or dynamic camera a third person view of the users AR session. The third person view may be synchronized with the user view and the virtual elements of the user view may be composited onto the third person view.

First claim

What is claimed is:

1. A computerized method, performed by a computing system associated with a camera having one or more hardware computer processors and one or more non-transitory computer readable storage device storing software instructions executable by the computing system to perform the computerized method comprising:
obtaining images of an environment that includes a mixed reality device as an environment video feed;
determining virtual elements rendered by the mixed reality device from a perspective of the mixed reality device; and
generating a composite video feed from the perspective of the camera that includes at least a portion of the virtual elements rendered by the mixed reality device and the environment video feed from the camera.

2. The method of claim 1 , wherein the environment video feed comprises images of physical objects within a physical environment.

3. The method of claim 2 , wherein the physical objects include a user of the mixed reality device.

4. The method of claim 1 , further comprising: transmitting the composite video feed to one or more display devices.

5. The method of claim 1 , wherein the virtual elements include a virtual avatar.

6. The method of claim 5 , wherein the virtual avatar interacts with a user of the mixed reality device by at least one of a prearranged routine, an interactive interaction, or a puppeteer.

7. The method of claim 1 , wherein the mixed reality device comprises a mixed reality headset worn by a user.

8. The method of claim 7 , further comprising: determining virtual elements rendered by a second mixed reality device from a second perspective of a second user; and rendering second virtual elements within a field of view of the camera from the perspective of the camera; wherein the composite video feed includes the rendered virtual elements and the rendered second virtual elements.

9. The method of claim 1 , wherein the camera is stationary.

10. The method of claim 1 , wherein the camera is moveable.

11. The method of claim 10 , wherein rendering of the virtual elements are updated as movements of the camera cause the perspective of the camera to update.

12. The method of claim 1 , further comprising: scanning the environment with an imaging device of the mixed reality device for a known image; determining an image location of the known image with reference to an environment coordinate system; and determining an origin point and gaze direction of the camera based on the determined image location.

13. The method of claim 12 , wherein the known image comprises a two-dimensional planar image.

14. The method of claim 1 , further comprising: receiving, from a localization device coupled to the camera, a real-time location information of the camera with reference to an environment coordinate system; and determining an origin point and gaze direction of the camera based on the received real-time location information.

15. The method of claim 14 , wherein the localization device comprises two or more imaging devices.

16. A computing system comprising: a hardware computer processor; and a non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by the hardware computer processor to cause the computing system to perform operations comprising:
obtaining images of an environment that includes a mixed reality device with a camera as an environment video feed;
determining virtual elements rendered by the mixed reality device from a perspective of the mixed reality device; and
generating a composite video feed from the perspective of the camera that includes at least a portion of the virtual elements rendered by the mixed reality device and the environment video feed from the camera.

17. The computing system of claim 16 , further comprising: transmitting the composite video feed to one or more display devices.

18. The computing system of claim 16 , wherein the virtual elements include a virtual avatar.

19. The computing system of claim 18 , wherein the virtual avatar interacts with a user of the mixed reality device by at least one of a prearranged routine, an interactive interaction, or a puppeteer.

20. The computing system of claim 16 , further comprising: scanning the environment with an imaging device of the mixed reality device for a known image; determining an image location of the known image with reference to an environment coordinate system; and determining an origin point and gaze direction of the camera based on the determined image location.

Assignees

• Magic Leap Inc

Inventors

• Miller Iv Thomas Marshall
• Kuribayashi Takashi
• Monos John

Classifications

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G02B27/0172 Primary

  characterised by optical features · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title