Smart content rendering on augmented reality systems, methods, and devices

Patent metadata

Field	Value
Publication number	US-12567217-B2
Application number	US-202318513160-A
Country	US
Kind code	B2
Filing date	Nov 17, 2023
Priority date	Dec 5, 2022
Publication date	Mar 3, 2026
Grant date	Mar 3, 2026

Abstract

A system and method for providing contextually-based content displays are provided. The system may include a device having a display providing content in a viewable region of the display. The device may include a camera device configured to track a scene of a real-world environment captured in a field of view of the camera device. The device may determine a region of interest in the scene. The device may perform object recognition on the scene tracked by the camera device. The device may determine an environmental interaction based on the object recognition and the region of interest. The device may adaptively alter, based on the environmental interaction, a position of the content provided by the display.

First claim

What is claimed:

1 . A device comprising:
a display providing content in a viewable region of the display; a camera device configured to track a scene of a real-world environment captured in a field of view of the camera device; one or more processors and a non-transitory memory including computer-executable instructions, which when executed, cause the device to at least:
determine a first region of interest in the scene;
perform object recognition on the scene tracked by the camera device;
determine an environmental interaction based on the object recognition and the first region of interest;
associate a second region of interest with the environmental interaction;
associate a third region of interest with the content in the viewable region of the display; and
adaptively alter, based on the environmental interaction, a position of the content provided by the display, wherein adaptively altering the position of the content provided by the display includes continuously moving the position of the content to reduce interference between the third region of interest and the second region of interest.

2 . The device of claim 1 , wherein the environmental interaction comprises at least one of:
an approaching object; an approaching person; a departing object; a departing person; an interaction with one or more objects; an interaction with one or more persons; or a gesture.

3 . The device of claim 1 , wherein the instructions, when executed by the one or more processors, further cause the device to:
determine a transition of the first region of interest from the display to the scene, or a transition of the first region of interest from the scene to the display; minimize or reduce a size of the content in an instance in which the first region of interest transitions from the display to the scene; and maximize or increase the size of the content in an instance in which the first region of interest transitions from the scene to the display.

4 . The device of claim 1 , wherein the device comprises an augmented reality device.

5 . The device of claim 4 , wherein the augmented reality device comprises a head-mounted device.

6 . The device of claim 1 , further comprising a second camera device configured to track a gaze of at least one eye, wherein the second camera device comprises a left eye tracking camera and a right eye tracking camera.

7 . The device of claim 1 , wherein the camera device comprises at least one outward-facing camera to track the scene.

8 . The device of claim 1 , wherein the display provides holographic content comprising augmented reality content.

9 . The device of claim 1 , wherein the scene captured in the field of view is associated with a gaze of at least one eye tracked by a second camera device.

10 . The device of claim 1 , wherein the instructions, when executed by the one or more processors, further cause the device to:
determine the first region of interest by determining a gaze depth of at least one eye or a gaze direction of the at least one eye captured by a second camera device.

11 . The device of claim 1 , wherein the object recognition identifies a person or an object in the scene.

12 . A method comprising:
performing object recognition on a scene of a real-world environment captured in a field of view of a camera device; determining a first region of interest in the scene; determining an environmental interaction based on the object recognition and the first region of interest; associating a second region of interest with the environmental interaction; associating a third region of interest with content in a viewable region of a display; and adaptively altering, based on the environmental interaction, a position of the content provided by the display, wherein adaptively altering the position of the content provided by the display includes continuously moving the position of the content to reduce interference between the third region of interest and the second region of interest.

13 . The method of claim 12 , further comprising:
determining the first region of interest based on a gaze of at least one eye tracked by a second camera device; determining, based on the gaze, a transition of the first region of interest between the display and the scene; minimizing or reducing a size of the content in an instance in which the first region of interest transitions from the display to the scene; and maximizing or increasing the size of the content in an instance in which the first region of interest transitions from the scene to the display.

14 . The method of claim 12 , further comprising:
applying one or more machine learning techniques to determine the environmental interaction based on training data associating one or more tracked gazes with one or more scenes.

15 . The method of claim 12 , further comprising:
performing, by a device, the adaptively altering the position of the content.

16 . A non-transitory, tangible computer-readable medium storing instructions that, when executed, cause:
performing object recognition on a scene tracked by a camera device; determining a first region of interest in the scene; determining an environmental interaction based on the object recognition and the first region of interest; associating a second region of interest with the environmental interaction; associating a third region of interest with content in a viewable region of a display; and adaptively altering, based on the environmental interaction, a position of the content provided on the display, wherein adaptively altering the position of the content provided by the display includes continuously moving the position of the content to reduce interference between the third region of interest and the second region of interest.

17 . The computer-readable medium of claim 16 , wherein the object recognition is performed continuously in real-time.

18 . The computer-readable medium of claim 16 , wherein the instructions, when executed, further cause: determining a region of interest corresponding to a gaze of at least one eye tracked by a second camera device, wherein the region of interest is based on a gaze depth of the at least one eye or a gaze direction of the at least one eye.

19 . The computer-readable medium of claim 16 , wherein the instructions, when executed, further cause:
moving, reducing, or minimizing the position of the content from the display in an instance in which an object of interest approaches at a predetermined threshold speed.

Assignees

• Meta Platforms Inc

Inventors

• Wu Shengzhi
• Faaborg Alexander

Classifications

• G06V10/25 Primary

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06V2201/07

  Target detection · CPC title

• G06T3/40

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title