Defining focal regions of images based on image metadata

Patent metadata

Field	Value
Publication number	US-12243259-B2
Application number	US-202217814064-A
Country	US
Kind code	B2
Filing date	Jul 21, 2022
Priority date	Jul 21, 2022
Publication date	Mar 4, 2025
Grant date	Mar 4, 2025

Abstract

An image processing system accesses an image depicting a particular object. The system identifies a location of the particular object within the image and defines, based on the identified location of the particular object within the image, a focal region of the image. The focal region can be used to modify the image for delivery and presentation on a webpage or software application.

First claim

We claim:

1. A computer-implemented method comprising:
receiving a request for displaying, in an image slot of a webpage, an image that depicts a particular object;
identifying a location of the particular object within the image;
defining, based on the identified location of the particular object within the image, a focal region of the image; and
serving a modified version of the image in the image slot of the webpage, wherein the modified version of the image contains an entirety of the focal region.

2. The method of claim 1 , wherein the image depicting the particular object is associated with metadata, and wherein the location of the particular object within the image is identified based on the metadata.

3. The method of claim 2 , wherein the metadata includes text related to the particular object, and wherein identifying the location of the particular object within the image includes:
analyzing the metadata using a natural language processing model that is configured to output a characteristic of the particular object based on the text,
wherein the characteristic of the particular object is used to identify the location.

4. The method of claim 3 , further comprising:
determining the particular object based on the metadata, wherein identifying the location of the particular object within the image includes identifying a portion of the image that matches the characteristic of the particular object.

5. The method of claim 1 , wherein identifying the location of the particular object within the image includes applying one or more object detection techniques to the image.

6. The method of claim 1 , wherein the request specifies an aspect ratio different from an aspect ratio of the image, and wherein the method further comprises: and
generating the modified version of the image having the specified aspect ratio and containing the entirety of the focal region.

7. The method of claim 1 , further comprising:
generating the modified version of the image based on a webpage theme of the webpage that includes the image slot.

8. The method of claim 1 , further comprising:
detecting a user access of the webpage, wherein the modified version of the image in the image slot of the webpage is served in response to the user access to the webpage.

9. The method of claim 1 , further comprising:
generating one or more modified images by cropping the image to a respective aspect ratio, wherein the focal region is fully contained within each of the modified images;
storing the one or more modified images; and
detecting a user access of the webpage that includes the image slot with a specified aspect ratio,
wherein serving the modified version of the image comprises serving a modified image of the one or more modified images based on a correspondence between the respective aspect ratio of the served modified image and the specified aspect ratio of the image slot.

10. The method of claim 1 , further comprising:
defining a bounding box around a portion of the image such that the focal region is fully contained within the bounding box; and
storing the definition of the bounding box in association with the image for use in cropping the image to a size of the bounding box.

11. The method of claim 1 , further comprising:
defining an initial window within the image for display in a viewport of a pannable image slot, the initial window containing at least a portion of the focal region; and
storing the definition of the initial window in association with the image.

12. The method of claim 1 , further comprising:
applying a zoom operation to the image to generate the modified version of the image to fill a bounding box; and
storing the modified version of the image.

13. The method of claim 1 , wherein identifying the location of the particular object within the image includes applying an object detection model to the image, and wherein the method further comprises:
sending for display to a user, a preview image that identifies the focal region;
receiving feedback from the user regarding a position of the focal region relative to the particular object; and
retraining the object detection model using the feedback.

14. A non-transitory computer readable storage medium storing executable instructions, execution of which by a processor causing the processor to:
receive a request for displaying, in an image slot of a webpage, an image that depicts a particular object;
identify a location of the particular object within the image;
define based on the identified location of the particular object within the image, a focal region of the image; and
serve a modified version of the image in the image slot of the webpage, wherein the modified version of the image contains an entirety of the focal region.

15. The non-transitory computer readable storage medium of claim 14 :
wherein the image depicting the particular object is associated with metadata;
wherein the location of the particular object within the image is identified based on the metadata;
wherein the metadata includes text related to the particular object; and
wherein identifying the location of the particular object within the image includes:
analyzing the metadata using a natural language processing model that is configured to output a characteristic of the particular object based on the text,
wherein the characteristic of the particular object is used to identify the location.

16. The non-transitory computer readable storage medium of claim 14 , wherein identifying the location of the particular object within the image includes applying one or more object detection techniques to the image.

17. The non-transitory computer readable storage medium of claim 14 , wherein:
the request specifies an aspect ratio different from an aspect ratio of the image; and
execution of the instructions further causes the processor to generate the modified version of the image having the specified aspect ratio and containing the entirety of the focal region.

18. The non-transitory computer readable storage medium of claim 14 , wherein execution of the instructions further causes the processor to:
generating one or more modified images by cropping the image to a respective aspect ratio, wherein the focal region is fully contained within each of the modified images;
storing the one or more modified images; and
detecting a user access of the webpage that includes the image slot with a specified aspect ratio,
wherein serving the modified version of the image comprises serving a modified image of the one or more modified images based on a correspondence between the respective aspect ratio of the served modified image and the specified aspect ratio of the image slot.

19. The non-transitory computer readable storage medium of claim 14 , wherein identifying the location of the particular object within the image includes applying an object detection model to the image, and wherein execution of the instructions further causes the processor to:
send for display to a user, a preview image that identifies the focal region;
receive feedback from the user regarding a position of the focal region relative to the particular object; and
retrain the object detection model using the feedback.

20. A system comprising:
at least one hardware processor; and
at least one non-transitory memory storing instructions, which, when executed by the at least one hardware processor, cause the system to:
receive a request for displaying, in an image slot of a webpage, an image

Assignees

• Shopify Inc

Inventors

• Leroux Stephan

Classifications

• G06N3/08

  Learning methods · CPC title

• G06F16/958

  Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking · CPC title

• G06V10/7788

  the supervisor being a human, e.g. interactive learning with a human teacher · CPC title

• G06V2201/10

  Recognition assisted with metadata · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title