Method and system for process monitoring

Patent metadata

Field	Value
Publication number	US-12450910-B2
Application number	US-202318334012-A
Country	US
Kind code	B2
Filing date	Jun 13, 2023
Priority date	Jun 13, 2023
Publication date	Oct 21, 2025
Grant date	Oct 21, 2025

Abstract

A method of monitoring an assembly process is disclosed herein. The method includes obtaining an event model for each of a plurality of objects in the assembly process with the event model for each of the plurality of objects including a predetermined time frame for a change in presence to occur. The method includes collecting an image sequence of the assembly process for monitoring and identifying if a change in presence for each of the plurality of objects occurred with a detector model. The method further includes reviewing the event model for each of the plurality of objects to determine if the predetermined time frame lapsed without the change in presence of a corresponding one of the plurality of objects being identified and issuing an alert if the predetermined time frame lapsed without the presence of a corresponding one of the plurality of objects being identified.

First claim

The invention claimed is:

1. A method of monitoring an assembly process, the method comprising:
generating an event model for each of a plurality of objects by:
receiving at least one training image sequence having a plurality of occurrences of the assembly process illustrating each of the plurality of objects,
identifying a presence of each of the plurality of objects in the at least one training image sequence, and
generating the event model corresponding to each of the plurality of objects based on a statistical prediction for when the object should appear in the training image sequence;

obtaining the event model for each of the plurality of objects in the assembly process, wherein the event model for each of the plurality of objects includes a predetermined time frame for a change in presence to occur;
collecting an image sequence of the assembly process for monitoring;
identifying if a change in presence for each of the plurality of objects occurred with a detector model;
reviewing the event model for each of the plurality of objects to determine if the predetermined time frame lapsed without the change in presence of a corresponding one of the plurality of objects being identified; and
issuing an alert if the predetermined time frame lapsed without the presence of the corresponding one of the plurality of objects being identified.

2. The method of claim 1 , wherein results from the detector model identifying the change in presence for each of the plurality of objects is filtered with a state machine based on if one of the plurality of objects was detected in at least one image proceeding or following.

3. The method of claim 1 , wherein identifying the change in presence for each of the plurality of objects includes determining a time when each of the plurality of objects either appeared or disappeared from the image sequence.

4. The method of claim 3 , wherein the time is based on a relative time set by the change in presence of one of the plurality of objects.

5. The method of claim 1 , wherein the plurality of objects includes multiple configurations of a single object.

6. The method of claim 1 , wherein collecting the image sequence of the assembly process occurs in real time.

7. The method of claim 1 , wherein
identifying the presence of each of the plurality of objects comprises utilizing the detector model to identify the presence of each of the plurality of objects in the at least one training image sequence.

8. The method of claim 1 , wherein the statistical prediction includes a predetermined time frame for the presence of the corresponding one of the plurality of objects.

9. The method of claim 8 , wherein the predetermined time frame is based on three times a standard deviation of a mean occurrence time for the presence of the corresponding one of the plurality of objects.

10. The method of claim 8 , wherein the predetermined time frame includes a relative time based on the presence of one of the plurality of objects.

11. The method of claim 1 , wherein the plurality of occurrences of the assembly process includes historical occurrences of the assembly process.

12. The method of claim 7 , wherein the detector model is trained by:
obtaining a separate training dataset corresponding to each of the plurality of objects with each of the separate training datasets including a set of tagged images identifying a corresponding one of the plurality of objects;
training parts-level detectors based on each of the separate training datasets; and
training the detector model based on each of the parts-level detectors.

13. The method of claim 12 , wherein each of the separate training datasets are created by:
receiving separate image sequences for each of the plurality of objects with a corresponding one of the plurality of objects identified in at least one image of the separate image sequence;
tracking each of the plurality of objects identified in the at least one image in a corresponding one of the separate image sequences;
tagging a region of interest in each image in the separate image sequences where a corresponding one of each of the plurality of objects was tracked; and
creating the separate training dataset for each of the plurality of objects by collecting the region of interest from each image in each of the separate image sequences where a corresponding object was tracked.

14. The method of claim 13 , wherein obtaining the separate training datasets corresponding to each of the plurality of objects includes eliminating false negative tags by verifying a presence of each of the plurality of objects in each of the separate training datasets against a ground-truth timeline for each of the plurality of objects.

15. A system for detecting objects in an assembly process, the system comprising:
at least one camera configured to capture a plurality of images; and
a controller configured to:
generate an event model for each of a plurality of objects by:
receiving at least one training image sequence having a plurality of occurrences of the assembly process illustrating each of the plurality of objects,
identifying a presence of each of the plurality of objects in the at least one training image sequence, and
generating the event model corresponding to each of the plurality of objects based on a statistical prediction for when the object should appear in the training image sequence;

obtain the event model for each of the plurality of objects in the assembly process, wherein the event model for each of the plurality of objects includes a predetermined time frame for a change in presence to occur;
collect an image sequence of the assembly process for monitoring;
identify if a change in presence for each of the plurality of objects occurred with a detector model;
review the event model for each of the plurality of objects to determine if the predetermined time frame lapsed without the change in presence of a corresponding one of the plurality of objects being identified; and
issue an alert if the predetermined time frame lapsed without the presence of the corresponding one of the plurality of objects being identified.

16. The system of claim 15 , wherein results from the detector model identifying the change in presence for each of the plurality of objects is filtered with a state machine based on if one of the plurality of objects was detected in at least one image proceeding or following.

17. The system of claim 15 , wherein
identifying the presence of each of the plurality of objects comprises utilizing the detector model to identify the presence of each of the plurality of objects in the at least one image sequence.

18. A non-transitory computer-readable storage medium embodying programmed instructions which, when executed by a processor, are operable for performing a method comprising:
generating an event model for each of a plurality of objects by:
receiving at least one training image sequence having a plurality of occurrences of an assembly process illustrating each of the plurality of objects,
identifying a presence of each of the plurality of objects in the at least one training image sequence, and
generating the event model corresponding to each of the plurality of objects based on a statistical prediction for when the object should appear in the training image sequence;

obtaining the event model for each of the plurality of objects in the assembly process, wherein the event model for each of the plurality of objects includes a predetermined time frame for a change in presence to occur;
collectin

Assignees

• Boeing Co

Inventors

• Chen Yang
• Khosla Deepak
• Courter Brandon Michael
• Arthur Shane Edward
• Howe Jeremy Daniel
• Heussy Anders Eirik

Classifications

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06V2201/07

  Target detection · CPC title

• G06V20/44 Primary

  Event detection · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06N20/00

  Machine learning · CPC title