Machine vision system for advancement of trailer loading/unloading visibility

Patent metadata

Field	Value
Publication number	US-12299627-B2
Application number	US-202217853748-A
Country	US
Kind code	B2
Filing date	Jun 29, 2022
Priority date	Jun 29, 2022
Publication date	May 13, 2025
Grant date	May 13, 2025

Abstract

Provided are embodiments for providing analytics indicative of object detection or fill-level detection at or near real-time based on video data captured during an unloading or loading process. A computerized system may detect and classify, using an object-detection machine learning (ML) model, an object based on the video data. A computerized system may further determine, using a fill-level ML model, a fill-level of the storage compartment based on a comparison of edges of the storage compartment to a total dimension corresponding to the edge. In this manner, the various implementations described herein provide a technique for computing systems employing image processing and machine learning techniques to a video data stream to generate analytics associated with the unloading or loading process at or near real-time.

First claim

What is claimed is:

1. At least one computer-storage media having computer-executable instructions embodied thereon that, when executed by a computing system having a processor and memory, cause the processor to perform operations comprising:
accessing a video indicative of the inside of a storage compartment from which items are being loaded or unloaded;
extracting video data for the video based at least on the items being loaded or unloaded from inside of the storage compartment to produce a plurality of video data features comprising a fill-level data feature and an object-detection data feature;
determining, using a fill-level machine learning (ML) model, a fill-level associated with the inside of the storage compartment based on at least the video and the fill-level data feature of the plurality of video data features;
classifying, using an object-detection ML model, an object detected inside of the storage compartment based at least on the video and the object-detection data feature of the plurality of video data features;
generating analytics indicative of the fill-level and the object; and
communicating the analytics to a display device to cause the display device to generate a graphical user interface (GUI).

2. The at least one computer-storage media of claim 1 , wherein the video is received from a monocular camera.

3. The at least one computer-storage media of claim 1 , wherein the GUI comprises a stream region that comprises a live stream of the video and a fill-level indication.

4. The at least one computer-storage media of claim 1 , wherein the GUI comprises an analytics region that comprises an indication of at least one of: a door open time, a door close time, a door open period, a human enter time, a human exit time, a maximum number of workers, a working hours, or a number of pallets.

5. The at least one computer-storage media of claim 1 , wherein the fill-level ML model is trained, using supervised machine learning, based on labeled data indicative of at least one of an edge, a length of the edge, an orientation of the edge, or an area coordinate associated with the storage compartment.

6. The at least one computer-storage media of claim 1 , wherein the object-detection ML model is trained, using supervised machine learning, based on labeled data indicative of at least one of a human object, a pallet object, a load stand object, a parcel retainer object, a parcel on a conveyer belt object, a forever bag object, the conveyer belt object, or a small container object.

7. The at least one computer-storage media of claim 1 , wherein:
the analytics indicative of the fill-level indicate the items stored in the storage compartment; and
the analytics indicative of the object indicate whether the object corresponds to a human object, a pallet object, a load stand object, a parcel retainer object, a parcel, a forever bag object, a conveyer belt object, or a small container object.

8. The at least one computer-storage media of claim 1 , wherein the object-detection data features comprises at least one of:
an object feature comprising at least one of: a human object, a pallet object, a load stand object, a parcel retainer object, a parcel, a forever bag object, a conveyer belt object, or a small container object; or
a status feature comprising at least one of: a door open status, a door closed status, a door partially open status, a door open but trailer not ready status, or an unloading or loading completed status.

9. The at least one computer-storage media of claim 1 , wherein the fill-level data features comprises at least one of: an edge detection, an edge length, an edge orientation, or area coordinates associated with the storage compartment.

10. A computerized system comprising:
at least one computer processor; and
computer memory storing computer-useable instructions that, when used by the at least one computer processor, cause the at least one computer processor to perform operations comprising:
accessing a video indicative of the inside of a storage compartment from which items are being loaded or unloaded;
extracting video data for the video based at least on the items being loaded or unloaded from inside of the storage compartment to produce a plurality of video data features comprising a fill-level data feature and an object-detection data feature;
determining a fill-level associated with the inside of the storage compartment based on at least the fill-level data feature of the plurality of video data features;
detecting and classifying an object inside of the storage compartment based at least on the object-detection data feature of the plurality of video data features; and
causing presentation of a graphical user interface (GUI) comprising:
a stream region comprising a stream of the video; and
an analytics region comprising analytics indicative of a progress in the items being loaded or unloaded from the storage compartment based on the fill-level and the object.

11. The computerized system of claim 10 , wherein causing the presentation of the GUI comprises:
updating the stream region to continue playing the stream of the video; and
removing an indication associated with the object when the object disappears from the stream of the video or adding another indication when another object is classified.

12. The computerized system of claim 10 , wherein the stream region comprises an indication associated with the object that comprises a visually distinct symbol based on classifying the object.

13. The computerized system of claim 10 , wherein an indication associated with the object is determined based on a Kalman filter that compares a current detected state of the object with a predicted state of the object, wherein the indication associated with the object corresponds to whether the object appears in the stream of the video.

14. The computerized system of claim 10 , wherein the video is received from a monocular camera positioned within or on a vehicle that comprises the storage compartment.

15. The computerized system of claim 10 , wherein the analytics indicative of the progress comprises an indication corresponding to at least one of: a door open status, a door closed status, a door partially open status, a door open but trailer not ready status, or an unloading or loading completed status.

16. The computerized system of claim 10 , wherein the object comprises at least one of: a human object, a pallet object, a load stand object, a parcel retainer object, a parcel on a conveyer belt object, a forever bag object, the conveyer belt object, or a small container object.

17. A computer-implemented method comprising:
accessing a fill-level machine learning (ML) model and an object-detection ML model;
training the fill-level ML model based on a first set of labeled data corresponding to a fill-level data feature, of a plurality of video data features, indicative of a fill-level of items being loaded or unloaded from a storage compartment;
validating the fill-level ML model based on a first set of unlabeled data applied to a first corresponding loss function;
training the object-detection ML model based on a second set of labeled data corresponding to an object-detection data feature, of the plurality of video data features, indicative of a classified object within the storage compartment;
validating the object-detection ML model based on a second set of unlabeled data applied to a second corresponding loss function; and
deploying the object-detection ML model and the fill-level ML model to a computing device.

1

Assignees

• United parcel service america inc

Inventors

• Choi Youngjun
• Lin Po-Nien
• Lee Hyunki

Classifications

• G06V10/70

  using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88) · CPC title

• B65G67/20

  Loading covered vehicles · CPC title

• G06V20/59

  inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions · CPC title

• G06V20/36

  Indoor scenes · CPC title

• G06Q10/109

  Time management, e.g. calendars, reminders, meetings or time accounting · CPC title