Displaying an augmented reality element listing supplemental items associated with a detected item

Patent metadata

Field	Value
Publication number	US-2025391073-A1
Application number	US-202418753870-A
Country	US
Kind code	A1
Filing date	Jun 25, 2024
Priority date	Jun 25, 2024
Publication date	Dec 25, 2025
Grant date	—

Abstract

A client device or an online system communicating with the device receives video data captured by a camera of the device, in which the video data depicts a field of view of a display area of the device. The device/system detects an object within the field of view based on the video data and applies one or more machine-learning algorithms to identify the object as an item available at a source. The device/system accesses item data for items available at the source and selects one or more supplemental items associated with the identified item based on item data for the identified item and each supplemental item. The device/system generates an augmented reality element including a listing of the supplemental item(s), as well as information or a selectable option associated with each supplemental item. The augmented reality element is then displayed in the display area of the device.

First claim

What is claimed is:

1 . A method, performed at a computer system comprising a processor and a computer-readable medium, comprising:
receiving video data captured by a camera of a client device, wherein the video data depicts a field of view of a display area of the client device; detecting an object within the field of view of the display area of the client device based at least in part on the video data; applying one or more machine-learning algorithms to the video data to identify the object as an item available at a source; accessing item data for a plurality of items available at the source; selecting one or more supplemental items associated with the identified item based at least in part on a set of item data for the identified item and each supplemental item of the one or more supplemental items; generating an augmented reality element comprising a listing of the one or more supplemental items and, for each supplemental item of the one or more supplemental items, one or more of: information associated with a corresponding supplemental item or a selectable option associated with the corresponding supplemental item; and displaying the augmented reality element in the display area of the client device, wherein the augmented reality element is overlaid onto a portion of the display area based at least in part on a location within the field of view of the display area at which the item is detected.

2 . The method of claim 1 , wherein applying the one or more machine-learning algorithms to the video data to identify the object as the item available at the source comprises:
accessing a database of items available at the source, wherein the database of items includes an image of each item available at the source; and applying the one or more machine-learning algorithms to the video data depicting the object and the image of each item available at the source to identify the object as the item available at the source.

3 . The method of claim 1 , wherein detecting the object within the field of view of the display area of the client device based at least in part on the video data comprises detecting the item within the field of view of the display area of the client device based on one or more of: a distance between the object and the client device, a position of the item within an additional augmented reality element displayed in the display area of the client device, or an interaction with the object by a user of the client device.

4 . The method of claim 1 , wherein selecting the one or more supplemental items associated with the identified item based at least in part on the set of item data for the identified item and each supplemental item of the one or more supplemental items comprises identifying the one or more supplemental items associated with the identified item based on one or more of: information describing a collection of items comprising the identified item and the one or more supplemental items, a pairing of the identified item with each supplemental item of the one or more supplemental items, or a promotion associated with the identified item and the one or more supplemental items.

5 . The method of claim 1 , wherein generating the augmented reality element comprising the listing of the one or more supplemental items and, for each supplemental item of the one or more supplemental items, information associated with the corresponding supplemental item comprises generating the augmented reality element comprising information describing a set of promotions associated with the corresponding supplemental item.

6 . The method of claim 1 , wherein generating the augmented reality element comprising the listing of the one or more supplemental items and, for each supplemental item of the one or more supplemental items, the selectable option associated with the corresponding supplemental item comprises generating the augmented reality element comprising the selectable option to display a set of directions to a location within the source associated with the corresponding supplemental item.

7 . The method of claim 6 , further comprising:
receiving a request to select the selectable option to display the set of directions to the location within the source associated with the corresponding supplemental item; retrieving information describing a current location of the client device within the source; retrieving information describing the location within the source associated with the corresponding supplemental item; determining the set of directions from the current location of the client device within the source to the location within the source associated with the corresponding supplemental item; generating an additional augmented reality element comprising the set of directions from the current location of the client device within the source to the location within the source associated with the corresponding supplemental item; and displaying the additional augmented reality element in the display area of the client device, wherein the augmented reality element is overlaid onto a portion of the display area.

8 . The method of claim 1 , wherein generating the augmented reality element comprising the listing of the one or more supplemental items and, for each supplemental item of the one or more supplemental items, the selectable option associated with the corresponding supplemental item comprises generating the augmented reality element comprising the selectable option to add the corresponding supplemental item to a shopping list.

9 . The method of claim 8 , further comprising:
receiving a request to select the selectable option to add the corresponding supplemental item to the shopping list; and adding the corresponding supplemental item to the shopping list.

10 . The method of claim 1 , wherein selecting the one or more supplemental items associated with the identified item based at least in part on the set of item data for the identified item and each supplemental item of the one or more supplemental items comprises:
retrieving a set of user data associated with a user of the client device; predicting a likelihood the user will perform an action associated with each supplemental item of a plurality of supplemental items associated with the identified item based at least in part on the set of user data associated with the user and the set of item data for a corresponding supplemental item; ranking the plurality of supplemental items based at least in part on the likelihood predicted for each supplemental item of the plurality of supplemental items; and selecting the one or more supplemental items associated with the identified item based at least in part on the ranking.

11 . A computer program product comprising a non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to perform steps comprising:
receiving video data captured by a camera of a client device, wherein the video data depicts a field of view of a display area of the client device; detecting an object within the field of view of the display area of the client device based at least in part on the video data; applying one or more machine-learning algorithms to the video data to identify the object as an item available at a source; accessing item data for a plurality of items available at the source; selecting one or more supplemental items associated with the identified item based at least in part on a set of item data for the identified item and each supplemental item of the one or more supplemental items; generating an augmented reality element comprising a listing of the one or more supplemental items and,

Assignees

• Maplebear Inc

Inventors

• Zhang Chao
• Han Bo
• Danshin Anton
• OuYang Yixi
• Dobaczewski Michal

Classifications

• G06V20/40

  in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title