Adaptive zoom try-on experience

Patent metadata

Field	Value
Publication number	US-12499483-B2
Application number	US-202318159509-A
Country	US
Kind code	B2
Filing date	Jan 25, 2023
Priority date	Jan 25, 2023
Publication date	Dec 16, 2025
Grant date	Dec 16, 2025

Abstract

Aspects of the present disclosure involve a system for automatically zooming in on objects in an image. The system accesses a video comprising a frame depicting a person and detects a fashion item being worn by the person in the frame of the video. The system, in response to detecting the fashion item being worn by the person, extracts, from the frame, a portion comprising a depiction of the fashion item being worn by the person. The system generates a focus image of the fashion item using the portion extracted from the frame and presents the focus image of the fashion item together with the video comprising the frame depicting the person.

First claim

What is claimed is:

1 . A method comprising:
accessing, by one or more processors of a device using a video access module, a video comprising a frame depicting a person; calculating, by an augmentation system via an application programming interface (API) server, characteristic points for a set of elements of the person to generate a mesh based on the calculated characteristic points; generating, by the augmentation system, one or more areas on the mesh of the person; aligning, by the augmentation system, the one or more areas of the person with one or more elements of a first augmented reality representation of a fashion item with a position; modifying, by the augmentation system, one or more visual properties of the one or more areas to cause the device to display the first augmented reality representation of the fashion item within the frame of the video at an individual display position relative to the display position of the person; detecting, by a fashion item detection module, the augmented reality representation of the fashion item being worn by the person in the frame of the video; in response to detecting the augmented reality representation of the fashion item being worn by the person, extracting, from the frame, a portion comprising a depiction of the augmented reality representation of the fashion item being worn by the person; generating, by a focus image generation module, a focus image of the augmented reality representation of the fashion item using the portion extracted from the frame to increase a size of the augmented reality representation of the fashion item in the focus image relative to the augmented reality representation of the fashion item depicted in the frame of the video, the focus image comprising a magnified view of the portion extracted from the frame comprising the depiction of the augmented reality representation of the fashion item being worn by the person; and presenting, by an image display module, the focus image of the augmented reality representation of the fashion item together with the video comprising the frame depicting the person.

2 . The method of claim 1 , wherein presenting the focus image comprises:
overlaying the focus image on the video at a predetermined position.

3 . The method of claim 1 , wherein presenting the focus image comprises:
displaying the focus image next to the video in a side-by-side arrangement.

4 . The method of claim 1 , further comprising:
receiving input that activates an augmented reality (AR) experience associated with trying on the fashion item; in response to the input, adding the augmented reality representation of the fashion item to a real-time video feed depicting the person captured by a camera of the device to generate the video; and tracking the augmented reality representation of the fashion item that has been added to the real-time video feed.

5 . The method of claim 1 , further comprising:
inferring intent of the person to view the focus image; and presenting the focus image in response to inferring the intent of the person to view the focus image.

6 . The method of claim 5 , further comprising:
detecting a gesture or movement made by the person depicted in the video; and determining that the gesture or movement causes the fashion item to be a subject of focus in the video, wherein the intent is inferred based on determining that the gesture or movement causes the augmented reality representation of the fashion item to be a subject of focus in the video.

7 . The method of claim 6 , wherein determining that the gesture or movement causes the fashion item to be a subject of focus in the video comprises determining that the gesture or movement causes the fashion item to be displayed closer to a center of a screen relative to a prior position on the screen.

8 . The method of claim 6 , further comprising:
controlling a size of the focus image that is presented together with the video based on determining that the gesture or movement causes the augmented reality representation of the fashion item to be the subject of focus in the video.

9 . The method of claim 8 , further comprising:
displaying the focus image in a first size in response to determining a first likelihood that the augmented reality representation of the fashion item is the subject of focus in the video; and as a further gesture or movement is made by the person depicted in the video, determining a second likelihood that the augmented reality representation of the fashion item is the subject of focus in the video, the second likelihood being greater than the first likelihood.

10 . The method of claim 9 , further comprising:
increasing the size of the focus image from the first size to a second size in response to determining the second likelihood that the augmented reality representation of the fashion item is the subject of focus in the video.

11 . The method of claim 6 , wherein the fashion item comprises earrings; and
wherein the gesture or movement comprises rotating a head of the person to bring an ear of the person into focus.

12 . The method of claim 6 , wherein the augmented reality representation of the fashion item comprises a watch worn on a wrist of the person; and
wherein the gesture or movement comprises rotating a wrist of the person to bring other portions of the wrist of the person into focus.

13 . The method of claim 1 , wherein the focus image comprises a zoomed image.

14 . The method of claim 1 , wherein the augmented reality representation of the fashion item comprises an augmented reality object.

15 . The method of claim 1 , wherein the focus image is presented in a round or circular window.

16 . A system comprising:
at least one processor of a device configured to perform operations comprising: accessing using a video access module a video comprising a frame depicting a person; calculating, by an augmentation system via an application programming interface (API) server, characteristic points for a set of elements of the person to generate a mesh based on the calculated characteristic points; generating, by the augmentation system, one or more areas on the mesh of the person; aligning, by the augmentation system, the one or more areas of the person with one or more elements of a first augmented reality representation of a fashion item with a position; and modifying, by the augmentation system, one or more visual properties of the one or more areas to cause the device to display the first augmented reality representation of the fashion item within the frame of the video at an individual display position relative to the display position of the person; detecting, by a fashion item detection module, the augmented reality representation of the fashion item being worn by the person in the frame of the video; in response to detecting the augmented reality representation of the fashion item being worn by the person, extracting, from the frame, a portion comprising a depiction of the augmented reality representation of the fashion item being worn by the person; generating, by a focus image generation module, a focus image of the augmented reality representation of the fashion item using the portion extracted from the frame to increase a size of the augmented reality representation of the fashion item in the focus image relative to the augmented reality representation of the fashion item depicted in the frame of the video, the focus image comprising a magnified view of the portion extracted from the frame comprising the depiction of the augmented reality representa

Assignees

• Snap Inc

Inventors

• Zohar Matan
• Berg Omri
• Sasson Gal
• Berger Itamar

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• H04N1/3871

  the composed originals being of different kinds, e.g. low- and high-resolution originals · CPC title

• H04N23/611

  where the recognised objects include parts of the human body · CPC title

• H04N1/3935

  with modification of image resolution, i.e. determining the values of picture elements at new relative positions · CPC title