Methods, apparatuses and computer program products for facilitating actions based on text captured by head mounted devices

Patent metadata

Field	Value
Publication number	US-2025182410-A1
Application number	US-202318529827-A
Country	US
Kind code	A1
Filing date	Dec 5, 2023
Priority date	Dec 5, 2023
Publication date	Jun 5, 2025
Grant date	—

Abstract

A system and method for determining interesting text to trigger actions of devices are provided. The system may include one or more head-mounted devices associated with a network. A head mounted device(s) may capture an image(s) and/or a video(s) corresponding to an environment detected in a field of view of a camera(s). The image(s) and/or the video(s) may include one or more text items associated with the environment. The head mounted device may determine whether a text item(s) of the one or more text items is interesting. The head mounted device may extract the text item(s) determined as being interesting and may superimpose the text item(s) at a position in the image(s) and/or the video(s). The head mounted device may trigger, based on the text item(s) determined as being interesting, one or more actions capable of being performed by the head mounted device.

First claim

What is claimed:

1 . A method comprising:
capturing, via a head mounted device, at least one image or at least one video corresponding to an environment detected in a field of view of at least one camera, wherein the at least one image or the at least one video comprises one or more text items associated with the environment; determining whether at least one text item of the one or more text items is interesting; extracting the at least one text item determined as being interesting and superimposing the at least one text item at a position in the at least one image or the at least one video; and triggering, based on the at least one text item determined as being interesting, one or more actions capable of being performed by the head mounted device.

2 . The method of claim 1 , wherein the head mounted device comprises smart glasses or an augmented or virtual reality device.

3 . The method of claim 1 , further comprising:
the determining whether the at least one text item of the one or more text items is interesting comprises determining that the at least one text item corresponds to at least one predetermined content item of text content designated as interesting associated with training data of one or more machine learning models.

4 . The method of claim 1 , further comprising:
determining that the at least one text item is interesting in response to determining, based on the at least one image or the at least one video, that at least one hand of a user holds, or points to, an object associated with the at least one text item.

5 . The method of claim 4 , further comprising:
cropping out one or more regions or other text items captured in the at least one image or the at least one video in response to determining the at least one hand of the user holds, or points to, the object associated with the at least one text item to obtain a second image or a second video comprising the at least one text item and excluding the one or more regions or the other text items.

6 . The method of claim 1 , further comprising:
determining, based on analyzing the at least one image or the at least one video, at least one region of interest associated with the at least one text item determined as being interesting and one or more background items or other items associated with a scene of the environment; and cropping out the one or more background items or the other items to obtain a second image or a second video comprising the at least one region of interest and the at least one text item.

7 . The method of claim 1 , wherein at least one action of the one or more actions comprises translating the at least one text item in a first language to a translated text item in a second language associated with the head mounted device.

8 . The method of claim 7 , further comprising:
performing the translating the at least one text item in the first language to the translated text item in the second language in response to detecting at least one finger of a user, associated with the head mounted device, pointing at or hovering over the at least one text item in the environment.

9 . The method of claim 7 , further comprising:
presenting, by the head mounted device, the translated text item in the second language superimposed within the at least one image or the at least one video.

10 . The method of claim 1 , further comprising:
generating a prompt, by the head mounted device, enabling a user associated with the head mounted device to select at least one action of the one or more actions to enable the head mounted device to perform the at least one action.

11 . An apparatus comprising:
one or more processors; and at least one memory storing instructions, that when executed by the one or more processors, cause the apparatus to:
capture, via the apparatus, at least one image or at least one video corresponding to an environment detected in a field of view of at least one camera, wherein the at least one image or the at least one video comprises one or more text items associated with the environment;
determine whether at least one text item of the one or more text items is interesting;
extract the at least one text item determined as being interesting and superimposing the at least one text item at a position in the at least one image or the at least one video; and
trigger, based on the at least one text item determined as being interesting, one or more actions capable of being performed by the apparatus.

12 . The apparatus of claim 11 , wherein the apparatus comprises a head mounted device, smart glasses or an augmented or virtual reality device.

13 . The apparatus of claim 11 , wherein when the one or more processors further execute the instructions, the apparatus is configured to:
perform the determine whether the at least one text item of the one or more text items is interesting by determining that the at least one text item corresponds to at least one predetermined content item of text content designated as interesting associated with training data of one or more machine learning models.

14 . The apparatus of claim 11 , wherein when the one or more processors further execute the instructions, the apparatus is configured to:
determine that the at least one text item is interesting in response to determining, based on the at least one image or the at least one video, that at least one hand of a user holds, or points to, an object associated with the at least one text item.

15 . The apparatus of claim 14 , wherein when the one or more processors further execute the instructions, the apparatus is configured to:
crop out one or more regions or other text items captured in the at least one image or the at least one video in response to determining the at least one hand of the user holds, or points to, the object associated with the at least one text item to obtain a second image or a second video comprising the at least one text item and excluding the one or more regions or the other text items.

16 . The apparatus of claim 11 , wherein when the one or more processors further execute the instructions, the apparatus is configured to:
determine, based on analyzing the at least one image or the at least one video, at least one region of interest associated with the at least one text item determined as being interesting and one or more background items or other items associated with a scene of the environment; and crop out the one or more background items or the other items to obtain a second image or a second video comprising the at least one region of interest and the at least one text item.

17 . The apparatus of claim 11 , wherein at least one action of the one or more actions comprises translating the at least one text item in a first language to a translated text item in a second language associated with the apparatus.

18 . The apparatus of claim 17 , wherein when the one or more processors further execute the instructions, the apparatus is configured to:
perform the translating the at least one text item in the first language to the translated text item in the second language in response to detecting at least one finger of a user, associated with the apparatus, pointing at or hovering over the at least one text item in the environment.

19 . A non-transitory computer-readable medium storing instructions that, when executed, cause:
capturing, via a head mounted device, at least one image or at least one video corresponding to an environment detected in a field of view of at least one camera, wherein th

Assignees

• Meta Platforms Inc

Inventors

• Zha Shengxin
• Sumbaly Roshan Rajesh
• Tan Jonathan

Classifications

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G02B27/017

  Head mounted · CPC title

• G06V40/107

  Static hand or arm · CPC title

• G06V30/10

  Character recognition · CPC title