Automatically detecting user-requested objects in digital images

Patent metadata

Field	Value
Publication number	US-12093306-B2
Application number	US-202318191651-A
Country	US
Kind code	B2
Filing date	Mar 28, 2023
Priority date	Jul 22, 2019
Publication date	Sep 17, 2024
Grant date	Sep 17, 2024

Abstract

The present disclosure relates to an object selection system that accurately detects and optionally automatically selects user-requested objects (e.g., query objects) in digital images. For example, the object selection system builds and utilizes an object selection pipeline to determine which object detection neural network to utilize to detect a query object based on analyzing the object class of a query object. In particular, the object selection system can identify both known object classes as well as objects corresponding to unknown object classes.

First claim

What is claimed is:

1. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
identifying a query that comprises a query object identification label indicating a query object to be detected in one or more digital images;
determining which path of a multi-path object detection pipeline comprising a first path for known objects and a second path for unknown objects to use for detecting the query object in the one or more digital images by:
selecting, based on determining whether the query object corresponds to a known object class or an unknown object class, an object class detection neural network from among a set of possible object class detection neural networks for classifying the query object, wherein the set of possible object class detection neural networks comprise a known object class detection neural network and an unknown object class detection neural network; and
utilizing the selected object class detection neural network to detect the query object within the one or more digital images.

2. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise:
detecting potential objects in the one or more digital images utilizing a region proposal neural network; and
generating approximate boundaries about the potential objects.

3. The non-transitory computer-readable medium of claim 2 , wherein:
the operations further comprise determining to use the first path for known objects based on determining that the query object corresponds to a known object; and
utilizing the selected object class detection neural network to detect the query object within the one or more digital images comprises generating object labels for the potential objects utilizing an object classification neural network.

4. The non-transitory computer-readable medium of claim 3 , wherein utilizing the selected object class detection neural network to detect the query object within the one or more digital images further comprises determining that an object label of one or more potential objects corresponds to a query object class associated with the query object.

5. The non-transitory computer-readable medium of claim 4 , wherein the operations further comprise generating an object mask for each detected instance of the query object utilizing an object mask model.

6. The non-transitory computer-readable medium of claim 3 , wherein the operations further comprise:
determining that an object label of at least one potential object does not correspond to the query object identification label; and
filtering out the at least one potential object based on the object label of the at least one potential object not corresponding to the query object identification label.

7. The non-transitory computer-readable medium of claim 2 , wherein:
the operations further comprise determining to use the second path for unknown objects based on determining that the query object does not correspond to a known object; and
utilizing the selected object class detection neural network to detect the query object within the one or more digital images comprises:
utilizing a concept embedding neural network to:
generate correlation scores between the potential objects relative to the query object identification label; and
select at least one potential object of the potential objects as an instance of the query object based on the correlation scores.

8. The non-transitory computer-readable medium of claim 7 , wherein utilizing the concept embedding neural network comprises generating image embeddings for each of the potential objects.

9. The non-transitory computer-readable medium of claim 8 , wherein utilizing the concept embedding neural network comprises generating a topic embedding for the query object identification label.

10. The non-transitory computer-readable medium of claim 9 , wherein the operations further comprise generating the correlation scores by comparing the topic embedding with the image embeddings.

11. A system comprising:
one or more memory devices; and
one or more processors coupled to the one or more memory devices that cause the system to perform operations comprising:
identifying a query that comprises a query object identification label indicating a query object to be detected in one or more digital images;
determining which path of a multi-path object detection pipeline comprising a first path for known objects and a second path for unknown objects to use for detecting the query object in the one or more digital images by:
selecting, based on determining whether the query object corresponds to a known object class or an unknown object class, an object class detection neural network from among a set of possible object class detection neural networks for classifying the query object, wherein the set of possible object class detection neural networks comprise a known object class detection neural network and an unknown object class detection neural network;
and utilizing the selected object class detection neural network to detect the query object within the one or more digital images.

12. The system of claim 11 , wherein:
the query comprises an image search request; and
the operations further comprise returning a subset of the one or more digital images that include the query object.

13. The system of claim 11 , wherein:
the operations further comprise determining to use the second path for unknown objects based on determining that the query object does not correspond to a known object; and
utilizing the selected object class detection neural network to detect the query object within the one or more digital images comprises:
detecting potential objects in the one or more digital images utilizing a region proposal neural network;
utilizing a concept embedding neural network to generate an image vector for each portion of the one or more digital images that includes a potential object;
utilizing the concept embedding neural network to generate a word vector for the query object;
generating correlation scores by measuring a similarity between each potential object vector and the word vector; and
detecting the query object in the one or more digital images based on the correlation scores.

14. The system of claim 13 , further comprising filtering out the potential objects with correlation scores below a similarity threshold prior to generating the correlation scores.

15. The system of claim 11 , wherein:
the operations further comprise determining to use the first path for known objects based on determining that the query object corresponds to a known object; and
utilizing the selected object class detection neural network to detect the query object within the one or more digital images comprises:
detecting objects in the one or more digital images utilizing a known object class detection neural network trained to locate objects in digital images that correspond to the query object identification label.

16. A method comprising:
identifying a query that comprises a query object identification label indicating a query object to be detected in one or more digital images;
determining which path of a multi-path object detection pipeline comprising a first path for known objects and a second path for unknown objects to use for detecting the query object in the one or more digital images by:
selecting, based on determining whether the query object corresponds to a known object class or an unknown object class, an object class detection neural n

Assignees

• Adobe Inc

Inventors

• Cohen Scott
• Lin Zhe
• Ling Mingyang

Classifications

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/2113

  by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation · CPC title