Independently determining adjustments to bounding shapes for detected objects in image data

Patent metadata

Field	Value
Publication number	US-12008792-B1
Application number	US-202117548380-A
Country	US
Kind code	B1
Filing date	Dec 10, 2021
Priority date	Dec 10, 2021
Publication date	Jun 11, 2024
Grant date	Jun 11, 2024

Abstract

Adjustments to bounding shapes for detected objects in image data may be independently determined. A bounding shape for an object detected in image data may be obtained. Independently determined adjustments for one or more edges of the bounding shape may be determined according to a multi-scale feature map generated from different resolutions of the image data that is provided as input to different dimension decoders to determine the adjustments to the bounding shape and respective confidence scores for the adjustments. The confidence scores are evaluated with respect to a confidence threshold to determine whether to provide the adjustments to the one or more edges of the bounding shape.

First claim

What is claimed is:

1. A system, comprising:
at least one processor; and
a memory, storing program instructions that when executed by the at least one processor, causes the at least one processor to:
receive a bounding box for an object detected in image data;
independently determine one or more adjustments to one or more edges of the bounding box according to a multi-scale feature map generated from different resolutions of the image data that is provided as input to different dimension decoders to determine the one or more adjustments to the bounding box and generate respective confidence scores for the one or more adjustments to the bounding box; and
determine that the confidence scores for the one or more adjustments to the one or more edges of the bounding box satisfies a confidence threshold; and
display an updated version of the bounding box according to the one or more adjustments to the one or more edges of the bounding box after the determination that the confidence scores for the one or more adjustments to the one or more edges of the bounding box satisfy the confidence threshold.

2. The system of claim 1 , wherein the memory stores further program instructions that when executed further cause the at least one processor to:
determine that a difference between the bounding box and a version of the bounding described by the one or more adjustments is above a difference threshold; and
wherein the display of the updated version of the bounding box according to the one or more adjustments to the one or more edges of the bounding box is performed after the determination that the difference between the bounding box and the version of the bounding box described by the one or more adjustments is above the difference threshold.

3. The system of claim 1 , wherein the bounding box is received as draw operation via a graphical user interface, and wherein the display of the updated version of the bounding box according to the one or more adjustments to the one or more edges of the bounding box automatically updates the drawn bounding box in the graphical user interface.

4. The system of claim 1 , wherein the at least one processor and the memory is implemented as part of a machine learning service offered by a provider network that displays the updated version of the bounding box via a labeling interface for the machine learning service.

5. A method, comprising:
obtaining, at an image labeling system, a bounding shape for an object detected in image data;
independently determining, by the image labeling system, one or more adjustments to one or more edges of the bounding shape according to a multi-scale feature map generated from different resolutions of the image data that is provided as input to different dimension decoders to determine the one or more adjustments to the one or more edges of the bounding shape and generate respective confidence scores for the one or more adjustments to the bounding shape; and
after determining that the respective confidence scores for the one or more adjustments to the one or more edges of the bounding shape satisfy a confidence threshold, providing, by the image labeling system, the one or more adjustments to the one or more edges of the bounding shape.

6. The method of claim 5 , further comprising:
determining that a difference between the bounding shape and a version of the bounding shape described by the one or more adjustments is above a difference threshold; and
wherein the providing of the one or more adjustments to the one or more edges of the bounding shape is performed after the determination that the difference between the bounding box and the version of the bounding shape described by the one or more adjustments is above the difference threshold.

7. The method of claim 6 , wherein the bounding shape for the object detected in image data is obtained via a validation interface.

8. The method of claim 5 , wherein the bounding shape is obtained as draw operation via a graphical user interface of the image labeling system, and wherein providing the one or more adjustments to the one or more edges of the bounding shape comprises automatically updating the drawn bounding shape in the graphical user interface with an updated version of the bounding shape according to the one or more adjustments to the one or more edges of the bounding shape.

9. The method of claim 5 , wherein another adjustment determined for another edge of the bounding shape different from the one or more edges is not provided after a determination that a confidence score for the other edge does not satisfy the confidence threshold.

10. The method of claim 5 , wherein independently determining the one or more adjustments to the one or more edges of the second bounding shape comprises:
cropping the image data, including the bounding shape for the detected object in the cropped image data;
applying a convolutional neural network (CNN) machine learning model to extract features of the cropped image data at the different resolutions;
combining the extracted features at the different resolutions to generate the multi-scale feature map;
applying vertical pooling and a one dimensional decoder to the multi-scale feature map to generate new left and right edges for the bounding shape;
applying horizontal pooling and another one dimensional decoder to the multi-scale feature map to generate new top and bottom edges for the bounding shape; and
wherein the new left and right edges and the new top and bottom edges are the one or more adjustments to the bounding shape.

11. The method of claim 5 , wherein the bounding shape is a polygon.

12. The method of claim 5 , wherein providing the one or more adjustments to the one or more edges of the bounding shape comprises displaying an updated version of the bounding shape determined according to the one or more adjustments to the one or more edges.

13. The method of claim 5 , wherein providing the one or more adjustments to the one or more edges of the bounding shape comprises storing the one or more adjustments to the one or more edges as a new version of the bounding shape as part of a training data set.

14. One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement:
obtaining a bounding shape for an object detected in image data;
independently determining one or more adjustments to one or more edges of the bounding shape according to a multi-scale feature map generated from different resolutions of the image data that is provided as input to different dimension decoders to determine the one or more adjustments to the bounding shape and generate respective confidence scores for the one or more adjustments to the bounding shape; and
determining that respective confidence scores for the one or more adjustments to the one or more edges of the bounding shape satisfies a confidence threshold; and
displaying the one or more adjustments to the one or more edges of the bounding shape after determining that the confidence scores for the one or more adjustments to the one or more edges of the bounding shape satisfy the confidence threshold.

15. The one or more non-transitory, computer-readable storage media of claim 14 , storing further instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement:
determining that a difference between the bounding shape and a version of the bounding shape described by the one or more adjustments is above a differ

Assignees

• Amazon Tech Inc

Inventors

• Bai Min
• Buck Jonathan
• Haffner Patrick Guy
• Chellapilla Kumar Hemachandra
• Li Li Erran

Classifications

• G06T7/13

  Edge detection · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/235 Primary

  based on user input or interaction · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06N3/08 Primary

  Learning methods · CPC title