Machine learning model for image forgery detection

Patent metadata

Field	Value
Publication number	US-12518141-B2
Application number	US-202318189044-A
Country	US
Kind code	B2
Filing date	Mar 23, 2023
Priority date	Mar 23, 2023
Publication date	Jan 6, 2026
Grant date	Jan 6, 2026

Abstract

Techniques for predicting whether a submission includes a forged image. A computer system receives a submission from a user that includes an image and image metadata, such as an identifier for the user and a User-Agent string value. An image pixel embedding is generated from the image, and a profile embedding is generated from the image metadata. The image embedding is indicative of whether the image is similar to known image forgeries. The profile embedding is generated from a user activity embedding indicative of User-Agent values associated with the user identifier. The profile embedding is generated using a machine learning model that uses stored parameters to associate user activity, device information, and forgery groups. The profile embedding thus indicates whether the user is associated with known image forgeries. The image pixel embedding and profile embedding are then used by a neural network to output a forgery prediction.

First claim

What is claimed is:

1 . A method, comprising:
receiving, at a computer system, a submission for authentication that includes an image and a user identifier for a user making the submission; and generating, by a prediction module within the computer system, a forgery prediction indicative of whether the image has been altered, wherein generating the forgery prediction includes:
generating an image pixel embedding from the image;
generating a profile embedding indicative of whether the user is associated with known image forgeries, wherein the profile embedding is generated by a machine learning model from a user activity embedding that is based on a) historical activity associated with the user identifier and b) a current User-Agent (UA) value associated with the submission, wherein learned parameters of the machine learning model have been determined using information convolution between 1) user activity information and device information, and 2) forgery group information and device information, wherein the user activity information, the device information, and the forgery group information are each expressed as UA values; and
outputting, by a neural network that receives the image pixel embedding and the profile embedding, the forgery prediction.

2 . The method of claim 1 , wherein the user activity embedding includes information about software entities associated with the user identifier that have previously made image submissions to the computer system.

3 . The method of claim 1 , wherein the user activity embedding is generated from User-Agent (UA) values indicative of user agents associated with the user identifier that have previously made image submissions to the computer system.

4 . The method of claim 1 , further comprising generating the user activity embedding by:
retrieving User-Agent (UA) values from a historical information table using the user identifier; and performing a text embedding function to transform the retrieved UA values into a numerical vector.

5 . The method of claim 4 , wherein generating the user activity embedding further includes:
accessing timestamps associated with the UA values; and using a neural network to encode sequence information indicative of when historical user agents associated with the UA values made submissions to the computer system.

6 . The method of claim 4 , wherein generating the user activity embedding further includes burst information in the user activity embedding, the burst information being indicative of the user's frequency of accesses made to the computer system.

7 . The method of claim 1 , wherein the machine learning model is operable to use the burst information as weights to indicate whether the user is behaving abnormally.

8 . The method of claim 1 , wherein the information convolution between user activity information and device information includes 1) propagating information that relates forgery group information and device information into user activity information, and 2) propagating user activity information into device information; and
wherein the information convolution between forgery group information and device information includes 1) propagating information from each forgery group to devices associated with those forgery groups, and 2) propagating device information and associated user activity into forgery group information.

9 . The method of claim 1 , wherein the prediction module is trained by:
traversing a set of training data having submissions including User-Agent (UA) values, wherein the traversing of a given submission in the set of training data includes generating a device embedding of a UA value for the given submission and inputting the device embedding to the machine learning model to generate learned parameters usable to associate user activity within image forgeries.

10 . The method of claim 9 , wherein the machine learning model uses a first graph, a second graph, and an initial forgery group embedding to generate the learned parameters, the first graph indicating relationships between user identifiers and associated UA values in the set of training data, the second graph indicating relationships between UA values and image forgery groups in the set of training data, and the initial forgery group embedding being generated from forgery groups identified in the set of training data.

11 . A non-transitory, computer-readable medium storing program instructions executable by a computer system to perform operations comprising:
receiving, from a particular software entity, a submission that includes an image and a user identifier; generating an image pixel embedding for the image; generating, from an indication of user activity associated with the user identifier, a profile embedding for the image, wherein the profile embedding is generated using a machine learning model that includes learned parameters that have been determined using information convolution between 1) user activity information and device information, and 2) forgery group information and device information, wherein the user activity information, the device information, and the forgery group information are each expressed as User-Agent (UA) values; and producing, by a neural network based on the image pixel embedding and the profile embedding, a forgery prediction for the image.

12 . The non-transitory, computer-readable medium of claim 11 , wherein the operations further comprise:
retrieving a set of entity identifiers associated with software entities that have made previous image submissions using the user identifier; generating, from the retrieved set of entity identifiers, a user activity embedding; and providing the user activity embedding to the machine learning model to obtain the profile embedding.

13 . The non-transitory, computer-readable medium of claim 12 , wherein the set of entity identifiers are UA values.

14 . The non-transitory, computer-readable medium of claim 13 , wherein the user activity embedding is generated to include sequence information indicative of a sequence of activity by UA values associated with the user identifier.

15 . The non-transitory, computer-readable medium of claim 14 , wherein the user activity embedding is generated to include burst information indicative of recent activity associated with the user identifier relative to historical activity.

16 . A system, comprising:
a processor subsystem; a memory subsystem storing program instructions executable by the processor subsystem to perform operations that include:
receiving, from a computing device, a digital data submission that includes:
digital data; and
metadata that includes a user identifier of a user associated with the digital data submission, and an entity identifier of a software entity that made the digital data submission on behalf of the user;

performing an analysis of the digital data submission using a machine learning model, wherein the performing includes:
generating, using the user identifier and the entity identifier, a first embedding indicative of a relationship between:
a first set or more entity identifiers that have previously been used to make digital data submissions to the system using the user identifier; and
a second set of one or more entity identifiers that have been associated with known instances of digital data having a particular digital data classification; and

generating, from the digital data itself, a second embedding; and

outputting, by a neural network, a prediction as to whether the digital data submission is in the particular digital data classifica

Assignees

• Paypal Inc

Inventors

• Chen Zhe
• Qi Panpan
• Zhang Jiazheng
• Zhang Jiyi
• Tang Quan Jin Ferdinand

Classifications

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06F21/31 Primary

  User authentication · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/045 Primary

  Combinations of networks · CPC title