Projector assisted augmented reality, adjusting AR size, shape and presentation based on real world space

Patent metadata

Field	Value
Publication number	US-12469224-B2
Application number	US-202117535522-A
Country	US
Kind code	B2
Filing date	Nov 24, 2021
Priority date	Nov 24, 2021
Publication date	Nov 11, 2025
Grant date	Nov 11, 2025

Abstract

Methods and systems for normalizing views of real world space and virtual environment of an interactive application as viewed through a pair of augmented reality glasses include receiving content of the interactive application for rendering on a display screen of the pair of augmented reality glasses. The content is streaming content provided in response to inputs provided by the user and includes virtual objects. The virtual objects are dynamically scaled to match a scale of the real world objects to a scale of the virtual objects. The content of the interactive application dynamically scaled is projected as an overlay over the real world objects of the real world space viewed through the pair of augmented reality glasses. The content that is dynamically scaled enables a normalized view between the real world space and the virtual environment.

First claim

The invention claimed is:

1 . A method, comprising:
receiving content of an interactive application for rendering on a display screen of a pair of augmented reality (AR) glasses worn by a user, the pair of AR glasses used to view real world space in vicinity of the user simultaneously with the content of the interactive application, the content being provided by the interactive application in response to inputs provided by the user and one or more other users during interaction with the interactive application, the content including virtual objects; dynamically scaling the virtual objects of the content of the interactive application to match to a scale of real world objects in the real world space viewed through the pair of AR glasses, wherein the dynamic scaling is performed by a machine learning algorithm by obtaining scanned images of the real world space and analyzing the scanned images to determine scale of the real world objects, in response to receiving content of the interactive application for rendering, the dynamic scaling of the virtual objects performed in accordance to the scale of the real world objects determined using the scanned images of the real world space; and presenting the virtual objects of the content of the interactive application dynamically scaled as an overlay over the real world objects of the real world space viewed through the pair of AR glasses, the scaled interactive content providing a normalized view of the interactive content when viewed alongside the real world objects, for the user.

2 . The method of claim 1 , wherein dynamically scaling further includes,
analyzing the scanned images to identify a layout of the real world objects in the real world space and to determine attributes of the real world objects, the attributes used to determine the scale of the real world objects in the real world space as viewed through the pair of AR glasses; and generating images of the virtual objects included in the content of the interactive application for projecting over the real world objects as overlay, the images of the virtual objects dynamically adjusted to match the scale of the virtual objects to the scale of the real world objects in the real world space as viewed through the pair of AR glasses prior to projecting as the overlay over the real world objects.

3 . The method of claim 2 , wherein the scanned images are received from a scanner in response to a first signal,
wherein the images of the virtual objects are dynamically adjusted using a projector in response to a second signal, and wherein the first signal and the second signal are triggered by the machine learning algorithm in response to receiving content of the interactive application for rendering, the machine learning algorithm analyzing the scanned images received in response to the first signal and generating the second signal to project the images, dynamically adjusted, over the real world objects.

4 . The method of claim 2 , wherein the attributes of a real world object determined from analyzing the scanned images include at least a position in the real world space, orientation in relation to another real world object, orientation in relation to the user interacting in the real world space, size of the real world object, and rendering distance of the real world object when viewed through the pair of AR glasses.

5 . The method of claim 1 , wherein presenting the content dynamically scaled further includes,
filtering out one or more virtual objects from the content prior to rendering the scaled interactive content, the filtering enabling viewing of the scaled interactive content alongside the real world space through the pair of AR glasses.

6 . The method of claim 1 , wherein presenting the content dynamically scaled includes providing a visual indicator to the virtual objects to indicate presence of the virtual objects included in the overlay.

7 . The method of claim 1 , wherein the dynamic scaling further includes scaling sound emanating from one or more virtual objects included in the content of the interactive application, the dynamic scaling of the sound performed based on context of the content of the interactive application.

8 . The method of claim 1 , wherein the interactive application is a video game played between a first player and second player who are remotely located, and wherein the content is gameplay content.

9 . The method of claim 8 , wherein the dynamic scaling includes scaling the gameplay content in accordance to context of a first set of real world objects in first real world space of the first player prior to overlaying the gameplay content over the first set of real world objects viewed through a first pair of AR glasses of the first player, and scaling the gameplay content in accordance to context of a second set of real world objects in second real world space of the second player prior to overlaying the gameplay content over the second set of real world objects viewed through a second pair of AR glasses of the second player.

10 . The method of claim 8 , further includes detecting a first selection of a first real world object from a first real world space in vicinity of the first player for inclusion as a first virtual object, and a second selection of a second real world object from a second real world space in vicinity of the second player for inclusion as a second virtual object,
wherein dynamic scaling includes scaling dimensions of the first virtual object and the second virtual object to match to the scale of the virtual objects of the content of the interactive application presented in the overlay, and wherein the first selection is initiated by the first player and the second selection is initiated by the second player.

11 . The method of claim 10 , wherein presenting content dynamically scaled includes,
including the first virtual object in the overlay rendered over the first real world space viewed via a first pair of AR glasses of the first player and including the second virtual object in the overlay rendered over the second real world space viewed via a second pair of AR glasses of the second player, the first and the second virtual objects rendered alongside the content of the interactive application, and the dynamic scaling normalizing view of the selected first virtual object to the first real world space and the selected second virtual object to the second real world space.

12 . The method of claim 8 , further includes,
detecting a first selection of a first real world object from a first real world space in vicinity of a first player for sharing with a second player, the first real world object is rendered as a first virtual object when rendered in the overlay of the content of the interactive application over the second real world space, wherein dynamic scaling includes scaling dimensions of the first virtual object to match to the scale of the real world objects in the second real world space viewed by the second player, and wherein the first real world object is selected for sharing by the first player.

13 . The method of claim 1 , wherein the dynamic scaling is done in response to a signal from a server executing the interactive application, or the interactive application, or a machine learning algorithm.

14 . A non-transitory, computer readable storage medium containing a computer program comprising computer executable instructions adapted to cause a computer system to perform a method of presenting content, comprising the steps of:
receiving content of an interactive application for rendering on a display screen of a pair of augmented reality (AR) glasses w

Assignees

• Sony Interactive Entertainment Inc

Inventors

• Alamdar Hadi
• Young Andrew
• Tokubo Todd

Classifications

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• A63F2300/8082

  Virtual reality · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• A63F13/52

  involving aspects of the displayed game scene · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title