Computer resource monitoring by processing images using a model comprising large language models

Patent metadata

Field	Value
Publication number	US-12360876-B2
Application number	US-202318195983-A
Country	US
Kind code	B2
Filing date	May 11, 2023
Priority date	May 11, 2023
Publication date	Jul 15, 2025
Grant date	Jul 15, 2025

Abstract

A system and a method for classifying images pertaining to computer resource usage of a user, using an ensemble of close ended questions and a neural network based language model. The method may be used to as aid to monitor and/or enforce computer usage policies in industrial organizations, government services, academy, education, and/or the like.

First claim

What is claimed is:

1. A method for classifying images by content, comprising:
acquiring a plurality of images pertaining to computer resource usage of a user;
detecting a plurality of patches pertaining to a textual content in the plurality of images;
extracting the textual content from the plurality of patches;
processing the textual content to generate a plurality of queries;
generating at least one compliance evaluation of the textual content to a usage policy by applying a model comprising a conversational language model on the plurality of queries; and
outputting a compliance measure of the computer resource usage of a user using at least one compliance evaluation;
wherein processing the textual content comprising:
accessing a storage to obtain a plurality of close ended questions; and
generating the plurality of queries each from a combination of the textual content and one of the plurality of close ended questions.

2. The method of claim 1 wherein the plurality of images are screen shots captured according to a timing policy.

3. The method of claim 1 wherein the model comprises obtaining at least one relevance evaluation corresponding to at least one subject by feeding a plurality of inferences generated by the conversational language into a decision model.

4. The method of claim 3 wherein the plurality of close ended questions pertaining to the at least one subject.

5. The method of claim 3 wherein the at least one subject comprising a white list of subjects allowed by the computer resource usage policy.

6. The method of claim 3 wherein the at least one subject comprising a black list of subjects restricted by the computer resource usage policy.

7. A system comprising a storage and at least one processing circuitry configured to:
acquire a plurality of images pertaining to computer resource usage of a user;
detect a plurality of patches pertaining to a textual content in the plurality of images;
extract the textual content from the plurality of patches;
process the textual content to generate a plurality of queries;

generate at least one compliance evaluation of the textual content to a usage policy by applying a model comprising a conversational language model on the plurality of queries; and
output a compliance measure of the computer resource usage of a user using at least one compliance evaluation;
wherein processing the textual content comprising:
accessing a storage to obtain a plurality of close ended questions; and
generating the plurality of queries each from a combination of the textual content and one of the plurality of close ended questions.

8. The system of claim 7 wherein the plurality of images are screen shots captured according to a timing policy.

9. The system of claim 7 wherein the model comprises obtaining at least one relevance evaluation corresponding to at least one subject by feeding a plurality of inferences generated by the conversational language into a decision model.

10. The system of claim 9 wherein the plurality of close ended questions pertaining to the at least one subject.

11. The system of claim 9 wherein the at least one subject comprising a white list of subjects allowed by the computer resource usage policy.

12. The system of claim 9 wherein the at least one subject comprising a black list of subjects restricted by the computer resource usage policy.

13. One or more computer program products comprising instructions for classifying images by content, wherein execution of the instructions by one or more processors of a computing system is to cause a computing system to:
acquire a plurality of images pertaining to computer resource usage of a user;
detect a plurality of patches pertaining to a textual content in the plurality of images;
extract the textual content from the plurality of patches;
process the textual content to generate a plurality of queries by at least:
accessing a storage to obtain a plurality of close ended questions, and
generating the plurality of queries each from a combination of the textual content and one of the plurality of close ended questions;

generate at least one compliance evaluation of the textual content to a usage policy by applying a model comprising a conversational language model on the plurality of queries; and
output a compliance measure of the computer resource usage of a user using at least one compliance evaluation.

Assignees

• Nec Corp America

Inventors

• Lev Tsvi

Classifications

• G06V10/764 Primary

  using classification, e.g. of video objects · CPC title

• G06F11/3438 Primary

  monitoring of user actions (tracking the activity of the user H04L67/535) · CPC title