Automated certificate systems and methods

Patent metadata

Field	Value
Publication number	US-12130610-B2
Application number	US-201816181194-A
Country	US
Kind code	B2
Filing date	Nov 5, 2018
Priority date	Nov 3, 2017
Publication date	Oct 29, 2024
Grant date	Oct 29, 2024

Abstract

The systems and methods provide an action recognition and analytics tool for use in manufacturing, health care services, shipping, retailing and other similar contexts. Machine learning action recognition can be utilized to determine cycles, processes, actions, sequences, objects and or the like in one or more sensor streams. The sensor streams can include, but are not limited to, one or more video sensor frames, thermal sensor frames, infrared sensor frames, and or three-dimensional depth frames. The analytics tool can provide for automatic creation of certificates for each instance of a subject product or service. The certificates can string together snippets of the sensor streams along with indicators of cycles, processes, action, sequences, objects, parameters and the like captured in the sensor streams.

First claim

What is claimed is:

1. An action recognition and analytics method comprising:
receiving a plurality of video frame streams from a plurality of manufacturing stations across an assembly line;
determining, in real time by artificial intelligence, one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects, and one or more parameters, in corresponding potions of the plurality of video frame streams, for each of a plurality of assembly items;
accessing a unique identifier of each of the plurality of assembly items;
mapping the unique identifier of the corresponding assembly item and the determined one of one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects, and one or more parameters of the corresponding assembly item to the corresponding portions of the plurality of video frame streams; and
for each of the plurality of assembly items, storing a certificate including the unique identifier of the corresponding assembly item, corresponding determined one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects, and one or more parameters, and corresponding portions of the plurality of video frame streams across the plurality of manufacturing stations.

2. The method according to claim 1 , wherein the unique identifier is generated using an algorithm based on the one of one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects, and one or more parameters for the corresponding assembly item.

3. The method according to claim 1 , wherein at least one of one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects, and one or more parameters can include the unique identifier of the corresponding assembly item.

4. The method according to claim 1 , wherein the unique identifier comprises a serial number of the corresponding assembly item.

5. The method according to claim 1 , wherein indicators of at least one of one or more cycles, one of one or more processes, one or more actions, one or more sequences, one or more objects, and one or more parameters are indexed to corresponding portions of the plurality of video frame streams by corresponding time stamps.

6. The method according to claim 1 , further comprising:
receiving one or more given indicators;
accessing one or more given data sets corresponding to one or more assembly items instance based on the one or more given indicators, wherein the one or more given data sets include one or more indicators of at least one of the one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects, and one or more parameters indexed to corresponding portions of the plurality of video frame streams for corresponding assembly items; and
outputting the corresponding portions of the plurality of video frame streams and the corresponding one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects, and one or more parameters for the corresponding one or more assembly items.

7. One or more non-transitory computing device-readable storage mediums storing instructions executable by one or more computing devices to perform an action recognition and analytics method comprising:
receiving one or more video frame streams from a plurality of manufacturing stations across an assembly line;
determining, in real time by convolution neural network deep learning one or more indicators of at least one of one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects, and one or more parameters, in the one or more video frame streams for each instance of a plurality of assembly items across each of the plurality of manufacturing stations;
accessing a unique identifier of each of the plurality of assembly items;
mapping the unique identifier of the corresponding portion of the one or more video frame streams and the one or more indicators of at least one of one or more cycles, one of one or more processes, one or more actions, one or more sequences, and one or more objects, and one or more parameters of the corresponding assembly item at the plurality of manufacturing stations across the assembly line to corresponding portions of the plurality of video frame streams; and
for each of the plurality of assembly items, storing a certificate including the unique identifier of the corresponding assembly item, corresponding determined one or more cycle, one or more processes, one or more actions, one or more sequences, one or more objects, and one or more parameters, and corresponding portions of the plurallity of video frame streams across the plurality of manufacturing stations.

8. The one or more non-transitory computing device-readable storage mediums storing instructions executable by one or more computing devices to perform the action recognition and analytics method according to claim 7 , wherein the unique identifier of the corresponding instance of the corresponding assembly item is received from a Manufacturing Execution System or a warehouse management system.

9. The one or more non-transitory computing device-readable storage mediums storing instructions executable by one or more computing devices to perform the action recognition and analytics method according to claim 7 , wherein the unique identifier comprises a serial number of the corresponding assembly item.

10. The one or more non-transitory computing device-readable storage mediums storing instructions executable by one or more computing devices to perform the action recognition and analytics method according to claim 7 , wherein the one or more indicators of at least one of one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects, and one or more parameters are indexed to corresponding portions of the one or more sensor streams by corresponding time stamps.

11. The one or more non-transitory computing device-readable storage mediums storing instructions executable by one or more computing devices to perform the action recognition and analytics method according to claim 7 , further comprising:
receiving a unique identifier of a given assembly item;
accessing one or more data structures to retrieve one or more given data sets corresponding to one or more assembly items based on one or more given indicators, wherein the one or more given data sets include the one or more indicators of at least one of one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects, and one or more parameters indexed to corresponding portions of the one or more video frame streams for corresponding assembly items; and
outputting the corresponding portions of the one or more video frame streams and the corresponding one or more indicators of the at least one of one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects, and one or more parameters for the given assembly item.

12. A system comprising:
one or more sensors including one or more video sensors; one or more non-transitory data storage units; and
one or more engines including one or more processing units executing instructions configured to:
receive a plurality of video frame streams;
determine, in real time one by artificial intelligence, one of one or more cycles, one or more processes, one or more actions, one or more sequences, one or more objects, and one or more parameters, in corresponding portions of the plurality of video frame streams, for a plur

Assignees

• R4N63R Capital Llc
• Apple Inc

Inventors

• Akella Prasad Narasimha
• Ashok Ananya Honnedevasthana
• Assoul Zakaria Ibrahim
• Chaudhury Krishnendu
• Gupta Sameer
• Narumanchi Sujay Venkata Krishna
• Prager David Scott
• Shankar Devashish
• Uggirala Ananth
• Chhabra Yash Raj

Classifications

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G09B19/00

  Teaching not covered by other main groups of this subclass (teaching or practice apparatus for gun-aiming or gun-laying F41G3/26) · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title