Computer-vision based process recognition

Patent metadata

Field	Value
Publication number	US-2016234464-A1
Application number	US-201514688230-A
Country	US
Kind code	A1
Filing date	Apr 16, 2015
Priority date	Feb 6, 2015
Publication date	Aug 11, 2016
Grant date	—

Abstract

A computer-vision based method for validating an activity workflow of a human performer includes identifying a target activity. The method includes determining an expected sequence of actions associated with the target activity. The method includes receiving a video stream from an image capture device monitoring an activity performed by an associated human performer. The method includes determining an external cue in the video stream. The method includes associating a frame capturing the external cue as a first frame in a key frame sequence. The method includes determining an action being performed by the associated human performer in the key frame sequence. In response to determining the action in the key frame sequence matching an expected action in the target activity, the method includes verifying the action as being performed in the monitored activity. In response to not determining the action in the key frame sequence, the method includes generating an alert indicating an error in the monitored activity.

First claim

What is claimed is:

1 . A computer-vision based method for validating an activity workflow of a human performer, the method comprising:
determining an expected sequence of actions associated with a target activity; receiving a video stream from an image capture device monitoring an activity performed by an associated human performer; determining an external cue in the video stream; associating a frame capturing the external cue as a first frame in a key frame sequence; determining an action being performed by the associated human performer in the key frame sequence; in response to determining the action in the key frame sequence matching an expected action in the target activity, verifying the action as being performed in the monitored activity; and, in response to not determining the action in the key frame sequence, generating an alert indicating an error in the monitored activity.

2 . The method of claim 1 , wherein the external cue is selected from a group consisting of:
an appearance of a certain object via object detection; a video analysis to detect a segment that includes lack of motion; a gesture made by the human performer being monitored; audio feedback; voice recognition; an input made at a GUI by the associated human performer; and, a combination of the above.

3 . The method of claim 1 further comprising:
searching for an object in the video stream;
performing a quantification about the object to determine a measure associated with the object;
embedding the measure in the video stream as metadata.

4 . The method of claim 3 , wherein the quantification includes:
generating a binary image of the object located in an image frame; and, summing pixels in the binary image to compute the measure.

5 . The method of claim 3 , wherein in response to an error being associated with the activity workflow, the method further comprising:
extracting the embedded measure from the video stream; comparing the extracted measure against a predetermined value; and, in response to a difference between the extracted measure and the predetermined value meeting and exceeding a predetermined threshold, generating an alert.

6 . The method of claim 3 , wherein the object is a syringe and the measure is a liquid fill-level of the syringe.

7 . The method of claim 3 , wherein the object is one of a bottle, container, and vial, and the measure is a number of items for delivery in the object.

8 . The method of claim 1 , where the target activity is a production of a pharmaceutical, and the action is a step of or relating to the target activity, wherein at least one action forms the target activity.

9 . The method of claim 1 , in response to generating the alert, providing the video stream to a user for review.

10 . The method of claim 1 , wherein the determining the action includes:
performing gesture recognition on the video stream; in response to detecting a gesture, segmenting a sequence of frames in the video stream proximate the detected gesture; and, searching the sequence of frames for the expected action.

11 . The method of claim 1 , wherein the determining the action includes:
searching for visual representations in the video sequence and determining if the representations correspond to a select object.

12 . A system for validating an activity workflow of a human performer, the system comprising a computer device including a memory in communication with a processor configured to:
determine an expected sequence of actions associated with a target activity; receive a video stream from an image capture device monitoring an activity performed by an associated human performer; determining an external cue in the video stream; associating a frame capturing the external cue as a first frame in a key frame sequence; determine an action being performed by the associated human performer in the key frame sequence; in response to determining the action in the key frame sequence matching an expected action in the target activity, verify the action as being performed in the monitored activity; and, in response to not determining the action in the key frame sequence, generate an alert indicating an error in the monitored activity.

13 . The system, of claim 12 , wherein the external cue is selected from a group consisting of:
an appearance of a certain object via object detection; a video analysis to detect a segment that includes lack of motion; a gesture made by the human performer being monitored; audio feedback; voice recognition; an input made at a GUI by the associated human performer; and, a combination of the above.

14 . The system, of claim 12 , wherein the processor is further configured to:
search for an object in the video stream; perform a quantification about the object to determine a measure associated with the object; and, embed the measure in the video stream as metadata.

15 . The system, of claim 14 , wherein the processor is further configured to:
generate a binary image of the object located in an image frame; and, sum pixels in the binary image to compute the measure.

16 . The system, of claim 14 , wherein the processor is further configured to:
in response to an error being associated with the activity workflow, extract the embedded measure from the video stream; compare the extracted measure against a predetermined value; and, in response to a difference between the extracted measure and the predetermined value meeting and exceeding a predetermined threshold, generate an alert.

17 . The system, of claim 12 , wherein the processor is further configured to:
in response to generating the alert, provide the video stream to a user for review.

18 . The system, of claim 12 , wherein the processor is further configured to:
perform gesture recognition on the video stream; in response to detecting a gesture, segment a sequence of frames in the video stream proximate the detected gesture; and, search the sequence of frames for the expected action.

19 . The system, of claim 12 , wherein the processor is further configured to:
search for visual representations in the video sequence and determining if the representations correspond to a select object.

20 . A computer-vision based method for validating an activity workflow of a human performer, the method comprising:
determining an expected sequence of actions associated with a target activity; receiving a video stream from an image capture device monitoring an activity performed by an associated human performer; determining an action performed by the associated human performer in the monitored video stream; in response to determining the action matching an expected action in the monitored video stream, verifying the action as being performed in the monitored activity; and, in response to not determining the action in the monitored video stream, generating an alert indicating an error in the monitored activity.

Assignees

• Xerox Corp

Inventors

• Loce Robert P
• Xu Beilei
• Bernal Edgar A
• Prabhat Saurabh
• Wu Wencheng

Classifications

• H04N7/183 Primary

  for receiving images from a single remote source · CPC title

• H04N5/23216

  Electricity · mapped topic

• H04N7/181 Primary

  for receiving images from a plurality of remote sources · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V20/40

  in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) · CPC title