Method for autonomous product detection and recognition on inventory structures within a store

Patent metadata

Field	Value
Publication number	US-12548345-B2
Application number	US-202519217166-A
Country	US
Kind code	B2
Filing date	May 23, 2025
Priority date	May 23, 2024
Publication date	Feb 10, 2026
Grant date	Feb 10, 2026

Abstract

One variation of a method includes: accessing an image of an inventory structure captured by a robotic system navigating within a facility; detecting an object occupying a slot in the inventory structure depicted in the image; extracting a set of visual features from the image; representing the set of visual features in a vector; projecting the vector into a multi-dimensional space populated template vectors representing product units of verified product types within the facility; calculating a similarity score between the set of visual features and template visual features represented in a cluster of template vectors in the multi-dimensional space, based on proximity between the vector and the cluster of template vectors; and, in response to the similarity score exceeding a threshold score, identifying the object as a product unit of a first product type affiliated with a first product identifier associated with the cluster of template vectors.

First claim

We claim:

1 . A method comprising:
during a first time period:
accessing a first image, in a set of images, of a first inventory structure captured by a robotic system while navigating within a facility;
detecting a first slot, in a set of slots, on the first inventory structure, depicted in the first image;
detecting a first product unit occupying the first slot depicted in a first region of the first image;
extracting a first set of visual features from the first region of the first image, the first set of visual features representing the first product unit;
detecting a first tag, proximal the first slot, in a second region of the first image;
identifying a first product identifier of a first product type of the first product unit based on features detected in the second region of the first image;
accessing a visual embedding model defined for the facility and trained to ingest a set of visual features of an image captured by the robotic system and automatically represent the set of visual features in a corresponding vector;
implementing the visual embedding model to represent the first set of visual features in a first template vector, in a set of template vectors, representing the first product unit;
labeling the first template vector with the first product identifier; and
populating a multi-dimensional space with the set of template vectors representing verified products within the facility; and
during a second time period succeeding the first time period:
accessing a second image of a second inventory structure captured by the robotic system;
detecting a second slot on the second inventory structure depicted in the second image;
detecting an object arranged in the second slot of the second inventory structure in the second image;
extracting a second set of visual features from the second image, the second set of visual features representing a packaging profile of the object;
representing the second set of visual features in a vector;
in response to the vector approximating a first cluster of template vectors in the multi-dimensional space, calculating a first similarity score between the second set of visual features represented in the vector and template visual features represented in the first cluster of template vectors; and
in response to the first similarity score exceeding a threshold score:
identifying the object as a product unit of a second product type affiliated with a second product identifier associated with the first cluster of template vectors;
assigning the second product identifier to the vector; and
storing the vector in the multi-dimensional space.

2 . The method of claim 1 , wherein accessing the first image of the first inventory structure captured by the robotic system comprises accessing the first image of the first inventory structure captured by the robotic system comprising:
a base; a drive system arranged in the base; a power supply; a set of mapping sensors; a processor configured to transform data collected by the set of mapping sensors into maps of a space surrounding the robotic system; a mast extending vertically from the base; a set of cameras arranged on the mast; and a wireless communication module configured to:
download waypoints and a master map of the facility from a remote computer system;
upload images captured by the set of cameras to the remote computer system; and
upload maps generated by the processor to the remote computer system.

3 . The method of claim 1 :
further comprising, during a scanning period preceding the first time period, dispatching the robotic system to autonomously navigate through the facility and record images of the inventory structures within the facility; and wherein accessing the first image of the first inventory structure comprises, at a computer system, receiving the first image of the first inventory structure from the robotic system.

4 . The method of claim 1 :
further comprising, during the first time period:
based on the first set of visual features and a first variance factor, calculating a second set of visual features representing a first synthetic variation of the first product unit;
representing the second set of visual features in a second template vector in the set of template vectors;
labeling the second template vector with the first product identifier;
based on the first set of visual features and a second variance factor, calculating a third set of visual features representing a second synthetic variation of the first product unit;
representing the third set of visual features in a third template vector in the set of template vectors; and
labeling the third template vector with the first product identifier; and
wherein populating the multi-dimensional space with the set of template vectors comprises populating the multi-dimensional space with the set of template vectors comprising the first template vector, the second template vector, and the third template vector.

5 . The method of claim 4 :
wherein extracting the first set of visual features comprises extracting the first set of visual features comprising a first packaging profile, a first set of text features, a first relative orientation, and a first lighting condition; wherein calculating the second set of visual features comprises calculating the second set of visual features comprising the first packaging profile, the first set of text features, the first relative orientation, and a second lighting condition; and wherein calculating the third set of visual features comprises calculating the third set of visual features comprising the first packaging profile, the first set of text features, a second relative orientation, and a third lighting condition.

6 . The method of claim 1 , wherein populating the multi-dimensional space with the set of template vectors comprises populating the multi-dimensional space with the set of template vectors comprising:
the first cluster of template vectors associated with the second product identifier of the second product type; a second cluster of template vectors associated with the first product identifier of the first product type; and a third cluster of template vectors associated with a third product identifier of a third product type.

7 . The method of claim 1 , further comprising, in response to the first similarity score falling below the threshold score:
rejecting the second product identifier for assignment to the vector; in response to the vector approximating a second cluster of template vectors in the multi-dimensional space, calculating a second similarity score between the second set of visual features represented in the vector and template visual features represented in the second cluster of template vectors; and in response to the second similarity score exceeding the threshold score:
identifying the object as a product unit of the first product type affiliated with the first product identifier associated with the second cluster of template vectors;
assigning the first product identifier to the vector; and
storing the vector in the multi-dimensional space.

8 . The method of claim 1 :
wherein calculating the first similarity score between the second set of visual features represented in the vector and template visual features represented in the first cluster of template vectors in response to the vector approximating the first cluster of template vectors in the multi-dimensional space comprises calculating the first similarity score between the second set of visual features represented in the vector and template visual features represented in the first cluster of template vectors in response to the vector approximating the first cluster of template vectors in

Assignees

• Simbe Robotics Inc

Inventors

• Bogolea Bradley
• Safi Jariullah
• Bahirat Kanchan

Classifications

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• B65G1/04

  mechanical · CPC title

• G06V10/762

  using clustering, e.g. of similar faces in social networks · CPC title

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title

• G01S17/89

  for mapping or imaging · CPC title