Method for scene segmentation

Patent metadata

Field	Value
Publication number	US-12430604-B2
Application number	US-202217839312-A
Country	US
Kind code	B2
Filing date	Jun 13, 2022
Priority date	May 19, 2016
Publication date	Sep 30, 2025
Grant date	Sep 30, 2025

Abstract

One variation of a method for segmenting scenes of product units arranged in inventory structures within a store includes: accessing an image based on data captured by a mobile robotic system; detecting a shelving segment in the image; reading a segment identifier from a segment tag, detected in the image, arranged on the shelving segment; accessing a product template representing a product type in the set of product types assigned to the shelving segment based on the segment identifier; detecting a set of product features, in the first region of the image. In response to detecting the set of product features analogous to features of the product template: confirming presence of the unit of the first product type on the shelf in the shelving segment and appending the first product type to a list of product types presently stocked in the shelving segment.

First claim

We claim:

1. A method for segmenting scenes of product units arranged in inventory structures within a store, the method comprising:
by a mobile robotic system:
autonomously navigating along shelving structures within the store; and
scanning shelving structures within the store during a first scan cycle; and

by a computer system:
accessing a first image generated based on data captured by the mobile robotic system autonomously traversing the store during the first scan cycle;
detecting, in the first image:
a first shelving segment;
a first shelf face of a first shelf in the first shelving segment;
a first vertical edge of the first shelf face;
a second vertical edge of the first shelf face; and
a second shelving segment adjacent the first shelving segment;
a second segment tag arranged on the second shelving segment;
a third shelving segment adjacent the first shelving segment and opposite the second shelving segment; and
a third segment tag arranged on the third shelving segment;

reading a second segment identifier from the second segment tag;
reading a third segment identifier from the third segment tag;
in response to detecting the first segment tag as obscured in the first image, interpolating the first segment identifier based on the second segment identifier and the third segment identifier;
based on the first segment identifier, accessing a first set of product templates representing a first set of product types assigned to the first shelving segment;
defining a first region in the first image depicting the first shelving segment based on the first vertical edge of the first shelf face and the second vertical edge of the first shelf face; and
in response to detecting a first set of product features, in the first region of the first image, analogous to a first product template, in the first set of product templates and corresponding to a first unit of a first product type in the first set of product types:
confirming presence of the first unit of the first product type on the first shelf in the first shelving segment; and
recording the first product type as presently stocked in the first shelving segment.

2. The method of claim 1 , further comprising, by the computer system:
in response to detecting a second set of product features, in the first region of the first image, analogous to a second product template, in the first set of product templates and corresponding to a second product type in the first set of product types:
confirming presence of a second unit of the second product type on the first shelf in the first shelving segment; and
recording the second product type as presently stocked in the first shelving segment.

3. The method of claim 1 , further comprising, by the computer system:
accessing a second image based on data captured by the mobile robotic system during an initial scan cycle preceding the first scan cycle in the store;
detecting the first shelving segment in the second image;
detecting, in the second image:
the first segment tag;
a first slot tag proximal the first shelf face;
a fourth shelf face of a fourth shelf below the first shelf face; and
a second slot tag proximal the fourth shelf face;

reading, from the first slot tag, a first product identifier corresponding to the first product type assigned to the first slot;
reading, from the second slot tag, a second product identifier corresponding to a second product type assigned to the second slot;
defining the first set of product types assigned to the first shelving segment, the set of product types comprising the first product type and the second product type; and
associating the first set of product types with the first segment identifier depicted on the first segment tag.

4. The method of claim 1 :
further comprising, by the computer system:
dispatching the mobile robotic system to a first position in the store adjacent the first shelving segment;
accessing a first set of images captured via a set of cameras mounted to the mobile robotic system, at the first position; and
stitching the first set of images captured by the mobile robotic system at the first position into a first composite image depicting the first shelving segment; and

wherein accessing the first image generated based on data captured by the mobile robotic system comprises accessing the first composite image.

5. The method of claim 1 , further comprising, by the computer system:
detecting, in the first image:
a third vertical edge of the second shelf face;
a fourth vertical edge of the second shelf face; and

based on the second segment identifier, accessing a second set of product templates representing a second set of product types assigned to the second shelving segment, the second set of product types different from the first set of product types;
defining a second region in the first region depicting the second shelving segment based on the third vertical edge of the second shelf face and the fourth vertical edge of the second shelf face; and
in response to detecting a second set of product features, in the second region of the first image, analogous to a second product template, in the second set of product templates and corresponding to a second product type in the second set of product types:
confirming presence of a second unit of the second product type on the second shelf in the second shelving segment; and
recording the second product as stocked in the second shelving segment.

6. The method of claim 5 :
wherein detecting the second shelf face in the first image comprises detecting the second shelf face right of the first shelf face in the first image;
wherein detecting the first vertical edge of the first shelf face comprises detecting a right vertical edge of the first shelf face;
wherein detecting the third vertical edge of the shelf face comprises detecting a left vertical edge of the shelf face;
further comprising, by the computer system, detecting:
a lateral offset between the right vertical edge of first shelf face and the left vertical edge of the second shelf face; and
a vertical offset between the first shelf face and the second shelf face; and

wherein detecting the second shelving segment in the first image comprises detecting the second shelving segment adjacent the first segment based on the lateral offset and the vertical offset.

7. The method of claim 1 :
further comprising, by the computer system:
detecting, in the first image, a first slot tag arranged on the first shelf face;
reading a first product identifier from the first slot tag;
retrieving, based on the first product identifier:
the first product type assigned to a first slot; and
a first slot boundary of the first slot; and

defining a first slot region of the first image based on the first slot boundary; and

wherein detecting the first set of product features analogous to the first product template corresponding to the first product type comprises detecting the first set of product features in the first slot region of the first image.

8. The method of claim 7 , further comprising, by the computer system:
reading a first product price from the first slot tag;
accessing a current product price from a product database based on the product identifier; and
in response to calculating a price difference between the first product price and the current product price greater than a threshold price difference, prompting a store associate to replace the first slot tag with a corrected slot tag.

9. The method of claim 7 :
wherein detecting the first slot tag in the first image comprises:
scanning the first image for slot tags based on a first slot tag boundary defining a first maximum tag dimens

Assignees

• Simbe Robotics Inc

Inventors

• Bahirat Kanchan
• Reddy Shiva
• Bogolea Bradley
• Safi Jariullah

Classifications

• G05D1/244

  using passive navigation aids external to the vehicle, e.g. markers, reflectors or magnetic means · CPC title

• G05D1/243

  Means capturing signals occurring naturally from the environment, e.g. ambient optical, acoustic, gravitational or magnetic signals (using passive navigation aids external to the vehicle G05D1/244; using signals from positioning sensors located off-board the vehicle G05D1/249) · CPC title

• G06V10/26

  Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion · CPC title

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title