Methods and systems for detecting errors in kit assembly

Patent metadata

Field	Value
Publication number	US-11042953-B1
Application number	US-201916724658-A
Country	US
Kind code	B1
Filing date	Dec 23, 2019
Priority date	Dec 23, 2019
Publication date	Jun 22, 2021
Grant date	Jun 22, 2021

Abstract

Methods and systems for packaging a product offering of product items associated with a merchant account. A product packaging model is associated with a kit of product items having a selected packaging option. During assembly, one or more images of the kit are captured and image features are detected and compared to parameters specified in the product packaging model. An error notification is output if a deviation is detected between the detected image features and the parameters prescribed by the product packaging model.

First claim

The invention claimed is:

1. A computer-implemented method for packaging a product offering of product items associated with a merchant account, the method comprising:
storing, in memory, a product packaging model associated with a kit for a merchant account, the product packaging model including parameters for two or more product items in the kit and a selected packaging option;
receiving one or more images of a kit during an assembly operation;
rendering the one or more images on a fulfillment partner device;
displaying an overlay on the one or more images on the fulfillment partner device, wherein the overlay includes data from the product packaging model;
detecting image features in the one or more images;
comparing the detected image features to the product packaging model and, based on the comparison, detecting a deviation; and
outputting a notification regarding the detected deviation.

2. The computer-implemented method of claim 1 , wherein the overlay includes a view of a three-dimensional model overlaid on the one or more images on the fulfillment partner device, wherein the three-dimensional model is rendered from the product packaging model.

3. The computer-implemented method of claim 2 , further comprising detecting registration of the three-dimensional model with a view of the kit in the one or more images and, in response, capturing an image and detecting the image features in the captured image.

4. The computer-implemented method of claim 1 , wherein detecting image features include processing the one or more images using one or more of edge detection, scale-invariant feature transform, thresholding, or Hough transform.

5. The computer-implemented method of claim 1 , wherein detecting a deviation includes comparing each of the detected image features to the parameters prescribed by the product packaging model and identifying that one of the detected image features is not included in the product packaging model.

6. The computer-implemented method of claim 1 , wherein detecting a deviation includes comparing each of the detected image features to the parameters prescribed by the product packaging model and identifying that one of the parameters in the product packaging model is not found in the detected image features.

7. The computer-implemented method of claim 1 , wherein detecting a deviation includes comparing each of the detected image features to the parameters prescribed by the product packaging model and determining that one of the detected image features deviates from its corresponding parameter in the product packaging model by more than a threshold amount.

8. The computer-implemented method of claim 1 , wherein outputting a notification include displaying the notification on the fulfillment partner device displaying at least one of the one or more images, and wherein the notification includes a visual indicator of the detected deviation.

9. The computer-implemented method of claim 1 , wherein outputting a notification includes transmitting a notification message to a device associated with the merchant account regarding the detected deviation.

10. The method claimed in claim 1 , wherein rendering includes operating a camera of the fulfilment partner device in a viewfinder mode and displaying the one or more images from the camera in the viewfinder mode.

11. A fulfillment partner device for packaging a product offering of product items associated with a merchant account, the fulfillment partner device comprising:
memory storing a product packaging model associated with a kit for a merchant account, the product packaging model including parameters for two or more product items in the kit and a selected packaging option;
a display screen;
a processor; and
a storage media storing computer-executable instructions that, when executed by the processor, are to cause the processor to:
receive one or more images of a kit during an assembly operation;
render the one or more images on the display screen;
display an overlay on the one or more images, wherein the overlay includes data from the product packaging model;
detect image features in the one or more images;
compare the detected image features to the product packaging model and, based on the comparison, detecting a deviation; and
output a notification regarding the detected deviation.

12. The fulfillment partner device of claim 11 , wherein the overlay includes a view of a three-dimensional model overlaid on the one or more images on the fulfillment partner device, wherein the three-dimensional model is rendered from the product packaging model.

13. The fulfillment partner device of claim 12 , wherein the instructions, when executed, are to further cause the processor to detect registration of the three-dimensional model with a view of the kit in the one or more images and, in response, capture an image and detect the image features in the captured image.

14. The fulfillment partner device of claim 11 , wherein the instructions, when executed, are to cause the processor to detect image features by processing the one or more images using one or more of edge detection, scale-invariant feature transform, thresholding, or Hough transform.

15. The fulfillment partner device of claim 11 , wherein the instructions, when executed, are to cause the processor to detect a deviation by comparing each of the detected image features to the parameters prescribed by the product packaging model and identifying that one of the detected image features is not included in the product packaging model.

16. The fulfillment partner device of claim 11 , wherein the instructions, when executed, are to cause the processor to detect a deviation by comparing each of the detected image features to the parameters prescribed by the product packaging model and identifying that one of the parameters in the product packaging model is not found in the detected image features.

17. The fulfillment partner device of claim 11 , wherein the instructions, when executed, are to cause the processor to detect a deviation by comparing each of the detected image features to the parameters prescribed by the product packaging model and determining that one of the detected image features deviates from its corresponding parameter in the product packaging model by more than a threshold amount.

18. The fulfillment partner device of claim 11 , wherein the instructions, when executed, are to cause the processor to output a notification by displaying the notification on the display screen, and wherein the notification includes a visual indicator of the detected deviation.

19. The fulfillment partner device of claim 11 , wherein the instructions, when executed, are to cause the processor to output a notification by transmitting a notification message to a device associated with the merchant account regarding the detected deviation.

20. A non-transitory computer-readable medium storing processor-executable instructions for packaging a product offering of product items associated with a merchant account, wherein the instructions, when executed by one or more processors, are to cause the one or more processors to:
store, in memory, a product packaging model associated with a kit for a merchant account, the product packaging model including parameters for two or more product items in the kit and a selected packaging option;
receive one or more images of a kit during an assembly operation;
render the one or more images on a fulfillment partner device;
display an overlay on the one or more images

Assignees

• Shopify Inc

Inventors

• Delgado Bryon Leonel
• Beauchamp Daniel
• Crowder Brian

Classifications

• G06F2113/20

  Packaging, e.g. boxes or containers · CPC title

• G06F30/17

  Mechanical parametric or variational design · CPC title

• G06T7/001

  using an image reference approach · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F30/20

  Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30) · CPC title