Unified model for accurate multi-object detection

Patent metadata

Field	Value
Publication number	US-2025217761-A1
Application number	US-202418402850-A
Country	US
Kind code	A1
Filing date	Jan 3, 2024
Priority date	Jan 3, 2024
Publication date	Jul 3, 2025
Grant date	—

Abstract

Examples provide a multi-object detection model for identifying different types of objects of interest using input images of a selected area including a plurality of objects. A multi-object detection manager identifies instances of each object type by adding indicators for identifying the different types of objects, such as, but not limited to, pallets, pallet tags, horizontal bars, vertical bars, wooden bases on pallets, and empty spaces. The indicators are provided within an overlay superimposed on the image. The indicators include text-based labels and/or non-text based color-coded indicators, such as color-coded bounding boxes. In such cases, instances of a first type of object are identified using a first indicator and instances of a second type of object are identified using a different second indicator. The labeled image including the indicators is generated to enable users to determine the locations of objects within a retail environment with greater accuracy and efficiency.

First claim

What is claimed is:

1 . A system for multi-object detection with improved accuracy, the system comprising:
an image capture device capturing an image of a plurality of objects of interest associated with a plurality of object types at a recognized location within a retail facility; and
a computer-readable medium storing instructions that are operative upon execution by a processor to:
analyze the image using a multi-object detection model identifying the plurality of objects of interest within the image; identify the plurality of objects of interest associated with the plurality of object types within the image, the plurality of objects of interest comprising a first object associated with a first object type and a second object associated with a second object type; and generate a labeled image of an area, the labeled image comprising a plurality of indicators within an overlay associated with the image, the plurality of indicators comprising a first indicator associated with the first object of interest within the image and a second indicator associated with the second object of interest within the image, wherein the labeled image is presented to a user via a user interface device.

2 . The system of claim 1 , wherein the instructions are further operative to:
train the multi-object detection model using labeled training data comprising labeled objects of interest associated with the plurality of object types.

3 . The system of claim 1 , wherein the plurality of indicators comprises a pallet indicator, a tag indicator, a wooden base indicator, a vertical bar indicator, a horizontal bar indicator, and a void space indicator.

4 . The system of claim 1 , wherein the instructions are further operative to:
generate a plurality of color-coded bounding boxes associated with the plurality of objects of interest, wherein all objects of a same object type within the image are enclosed within a bounding box of a same color.

5 . The system of claim 1 , wherein the instructions are further operative to:
enclosing a first set of objects from the plurality of objects of interest within a first set of bounding boxes of a first color, the first set of bounding boxes associated with a first object type; enclosing a second set of objects from the plurality of objects of interest within a second set of bounding boxes of a second color, the second set of bounding boxes associated with a second object type; and enclosing a third set of objects from the plurality of objects of interest within a third set of bounding boxes of a third color, the third set of bounding boxes associated with a third object type.

6 . The system of claim 5 , wherein the instructions are further operative to:
enclosing a fourth set of objects from the plurality of objects of interest within a fourth set of bounding boxes of a fourth color, the fourth set of bounding boxes associated with a fourth object type; and enclosing a fifth set of objects from the plurality of objects of interest within a fifth set of bounding boxes of a fifth color, the fifth set of bounding boxes associated with a fifth object type.

7 . The system of claim 1 , wherein the instructions are further operative to:
map the plurality of objects associated with the plurality of object types to the recognized location in an item-to-location mapping table using labeled image data associated with the labeled image.

8 . A method for multi-object detection, the method comprising:
obtaining an image of a recognized area associated with a retail facility, the image comprising a plurality of objects of interest associated with a plurality of object types; analyzing the image by a multi-object detection model, the multi-object detection model trained to recognize the plurality of object types using image data; identifying the plurality of objects of interest associated with the plurality of object types within the image and an object type for each identified object; generating a plurality of indicators within the image data associated with the plurality of objects of interest, the plurality of indicators comprising a first indicator of a first object type associated with a first object of a first object type and a second indicator of a second object type associated with a second object of a second object type in the plurality of object types, wherein the second indicator is a different indicator than the first indicator; generating a labeled image of the recognized area, the labeled image comprising the plurality of indicators within an overlay associated with the image; and presenting the labeled image to a user via a user interface device.

9 . The method of claim 8 , further comprising:
training the multi-object detection model to recognize the plurality of object types using labeled training data comprising labeled objects of interest associated with the plurality of object types.

10 . The method of claim 9 , wherein the plurality of object types comprises pallets, pallet tags, pallet wooden bases, pallet steel vertical bars, pallet steel horizontal bars, pallet void spaces, and pallet partial-empty spaces.

11 . The method of claim 8 , further comprising:
generating a first bounding box of a first color enclosing each instance of a first type of object in the image; and generating a second bounding of a second color enclosing each instance of a second type of object in the image.

12 . The method of claim 8 , further comprising:
generating a plurality of labels within the overlay corresponding to the plurality of objects of interest, the plurality of labels comprising a first label associated with each instances of a first type of object within the image and a second label associated with each instance of a second type of object within the image, wherein the first label comprises text identifying the first type of object, and wherein the second label comprises text identifying the second type of object.

13 . The method of claim 8 , further comprising:
enclosing each pallet object and each pallet tag object with a rectangular bounding box enclosing each pallet object within the image; and enclosing each pallet wooden base object, each pallet steel vertical bar object, and each pallet steel horizontal bar object with a polygon bounding box during image analysis, wherein each bounding box associated with each different type of object is a different color.

14 . The method of claim 8 , further comprising:
mapping the plurality of objects associated with the plurality of object types to a recognized location in an item-to-location mapping table using the image data associated with the labeled image.

15 . One or more computer storage devices having computer-executable instructions stored thereon, which, upon execution by a computer, cause the computer to perform operations comprising:
training a multi-object detection model using labeled training data comprising labeled objects of interest associated with a plurality of object types; analyzing an image from a plurality of images using a multi-object detection model identifying a plurality of objects of interest within the image; identifying the plurality of objects of interest associated with the plurality of object types within the image, the plurality of objects of interest comprising a first object associated with a first object type and a second object associated with a second object type; generating labeled image data based on the image, the labeled image data comprising a plurality of indicators associated with the image, the plurality of indicators comprising a first indicator associ

Assignees

• Walmart Apollo Llc

Inventors

• Ghosh Ashlin
• Lobo Paul
• Duan Zhaoliang
• Balusu Raghava
• Jade Avinash Madhusudanrao
• Rader Eric W
• Zhang Lingfeng
• Yuan Mingquan
• Chakraborty Siddhartha
• Zhang Han
• Robinson William Craig
• Ellison Benjamin
• Arora Ishan
• Chen Yilun
• Shah Ketan
• Pachauri Abhinav
• Chai Rongdong

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V2201/07

  Target detection · CPC title

• G06Q10/087 Primary

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V2201/10

  Recognition assisted with metadata · CPC title