Utilizing object attribute detection models to automatically select instances of detected objects in images

Patent metadata

Field	Value
Publication number	US-11468550-B2
Application number	US-201916518850-A
Country	US
Kind code	B2
Filing date	Jul 22, 2019
Priority date	Jul 22, 2019
Publication date	Oct 11, 2022
Grant date	Oct 11, 2022

Abstract

The present disclosure relates to an object selection system that accurately detects and automatically selects target instances of user-requested objects (e.g., a query object instance) in a digital image. In one or more embodiments, the object selection system can analyze one or more user inputs to determine an optimal object attribute detection model from multiple specialized and generalized object attribute models. Additionally, the object selection system can utilize the selected object attribute model to detect and select one or more target instances of a query object in an image, where the image includes multiple instances of the query object.

First claim

What is claimed is:

1. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computing device to:
based on identifying a digital image comprising a plurality of detectable objects, receive user input from a client device comprising a selection query indicating a target query object to be selected within the digital image;
analyze the selection query to identify a first detection input comprising a query object and a second detection input comprising an object attribute corresponding to the query object;
based on the first detection input, detect multiple query object instances in the digital image from the plurality of detectable objects utilizing an object detection neural network;
based on the second detection input, select, from a plurality of object attribute detection models designed to identify respective object attributes, an object attribute detection model specifically trained to identify the object attribute indicated by the selection query;
detect that a first query object instance from the multiple query object instances is the target query object by utilizing the object attribute detection model to determine that the first query object instance reflects the object attribute by analyzing object masks for the multiple query object instances to identify which of the object masks include the object attribute by determining that a first relative position for the first query object instance corresponds to the object attribute based on comparing a first center position of the first query object instance to a second center position of a second query object instance of the multiple query object instances to identify the first relative position for the first query object instance; and
provide, to the client device, the digital image with the target query object selected in response to the selection query.

2. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to parse the selection query from the client device to determine:
a noun that identifies the query object; and
an adjective that identifies the object attribute.

3. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
determine, from the object attribute of the selection query, an indicated position of the query object within the digital image; and
detect that the first query object instance is the target query object by:
determining positions within the digital image for each of the multiple query object instances utilizing an object position attribute detection model; and
detecting the first query object instance is the target query object based on the first query object instance having a position in the digital image that is closest to the indicated position.

4. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
determine, from the object attribute of the selection query, an indicated relative object position of the query object within the digital image; and
detect the first query object instance is the target query object based on the first relative position for the first query object instance corresponding to the indicated relative object position.

5. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
determine, from the object attribute of the selection query, an indicated object depth of the query object within the digital image; and
detect the first query object instance is the target query object by:
generating a depth map of the digital image utilizing a depth map neural network;
identifying, based on the depth map, a first positional depth of the first query object instance utilizing the depth map neural network; and
detecting the first query object instance is the target query object based on the first positional depth of the first query object instance corresponding to the indicated object depth.

6. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
determine, from the object attribute of the selection query, an indicated facial expression of the query object within the digital image; and
detect the first query object instance is the target query object by determining that the first query object instance reflects the indicated facial expression.

7. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
determine, from the object attribute of the selection query, an object shape of the query object within the digital image; and
detect the first query object instance is the target query object by identifying the first query object instance corresponding to the object shape.

8. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
determine, from the object attribute of the selection query, an indicated color of the query object within the digital image; and
detect the first query object instance is the target query object by:
identifying a color for each of the multiple query object instances in the digital image utilizing an object color attribute detection model; and
determining, based on comparing the color identified for each of the multiple query object instances to the indicated color, that the first query object instance has a greater correspondence to the indicated color than other instances of the multiple query object instances.

9. The non-transitory computer-readable medium of claim 8 , further comprising instructions that, when executed by the at least one processor, cause the computing device to identify the color for the first query object instance utilizing the object color attribute detection model by comparing one or more pixels of the first query object instance to the indicated color in a multi-dimensional color space.

10. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
determine that the object attribute does not correspond to a known object attribute type; and
detect, based on determining that the object attribute does not correspond to a known object attribute type, the first query object instance is the target query object by:
generating tags for one or more of the multiple query object instances utilizing a tagging neural network; and
matching the object attribute with a tag generated for the first query object instance.

11. The non-transitory computer-readable medium of claim 10 , wherein the instructions, when executed by the at least one processor, cause the computing device to detect the first query object instance is the target query object by filtering out one or more other query object instances of the multiple query object instances based on the one or more other query object instances having tags not corresponding to the object attribute.

12. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
determine, from the object at

Assignees

• Adobe Inc

Inventors

• Cohen Scott
• Lin Zhe
• Ling Mingyang

Classifications

• G06F16/5838

  using colour · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title

• G06T2210/12

  Bounding box · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title