Systems and user interfaces for dynamic and interactive investigation based on automatic clustering of related data in various data structures

Patent metadata

Field	Value
Publication number	US-10721268-B2
Application number	US-201916239081-A
Country	US
Kind code	B2
Filing date	Jan 3, 2019
Priority date	Mar 15, 2013
Publication date	Jul 21, 2020
Grant date	Jul 21, 2020

Abstract

In various embodiments, systems, methods, and techniques are disclosed for generating a collection of clusters of related data from a seed. Seeds may be generated based on seed generation strategies or rules. Clusters may be generated by, for example, retrieving a seed, adding the seed to a first cluster, retrieving a clustering strategy or rules, and adding related data and/or data entities to the cluster based on the clustering strategy. Various cluster scores may be generated based on attributes of data in a given cluster. Further, cluster metascores may be generated based on various cluster scores associated with a cluster. Clusters may be ranked based on cluster metascores. Various embodiments may enable an analyst to discover various insights related to data clusters, and may be applicable to various tasks including, for example, tax fraud detection, beaconing malware detection, malware user-agent detection, and/or activity trend detection, among various others.

First claim

What is claimed is:

1. A computer system comprising:
one or more computer readable storage devices configured to store:
host-based events associated with one or more computing devices; and
activity trend-related data items; and

one or more hardware computer processors in communication with the one or more computer readable storage devices and configured to execute computer executable instructions to cause the computer system to:
execute a cluster engine configured to at least:
determine a first group of host-based events that indicate a same first activity type and are associated with a first host and a reference time period;
determine, based at least on the first group of host-based events, a first statistical deviation in the first activity type on the first host for the reference time period;
determine a second group of host-based events the indicate the same first activity type and are associated with the first host and a test time period;
determine, based at least on the second group of host-based events, a second statistical deviation in the first activity type on the first host for the test time period;
in response to determining that the first statistical deviation compared to the second statistical deviation satisfies a particular threshold, designate a host-based event from the second group as a seed;
generate a data item cluster based on the seed, wherein generating the data item cluster comprises:
adding the seed to the data item cluster; and
adding to the data item cluster one or more activity trend-related data items, from the activity trend-related data items, determined to be associated with the seed; and
determine scores for the data item cluster and a plurality of additional data items clusters generated based on host-based events; and

execute a workflow engine configured to at least:
cause presentation of the data item cluster and the plurality of additional data item clusters in a user interface of a client computing device; and
order the presented data item cluster and the plurality of additional data item clusters in the user interface based at least in part on the respective determined scores for the data item cluster and the plurality of additional data item clusters.

2. The computer system of claim 1 , wherein the activity trend-related data items include at least one of: data items associated with captured host-based events, Internet Protocol addresses, external domains, users, or computing devices, and wherein hosts comprise computing devices in a network.

3. The computer system of claim 1 , wherein the one or more hardware computer processors are configured to execute the computer executable instructions to further cause the computer system to:
execute the cluster engine further configured to at least:
identify the one or more activity trend-related data items determined to be associated with the seed based at least on a clustering strategy, wherein the clustering strategy queries the host-based events and/or the activity trend-related data items to determine at least one of: the particular host associated with the seed, one or more host-based events associated with the particular host, one or more host-based events associated with the seed, users of the particular host, data items associated with the particular host, other hosts associated with the same particular activity type of host-based events, Internet Protocol addresses associated with the particular host, external domains associated with the seed, or computing devices associated with the particular host.

4. The computer system of claim 3 , wherein identifying one or more activity trend-related data items determined to be associated with the seed further comprises determining a particular activity trend-related data item and the seed are both associated with a common metadata property value.

5. The computer system of claim 4 , wherein the common property value includes at least one of: a username, a domain, an Internet Protocol address, a computing device identifier, or an event identifier.

6. The computer system of claim 1 , wherein the first statistical deviation comprises a Z-score.

7. A computer-implemented method comprising:
by one or more processors executing program instructions:
executing a cluster engine configured to at least:
access one or more computer readable storage devices configured to store:
host-based events associated with one or more computing devices; and
activity trend-related data items;
determine a first group of host-based events that indicate a same first activity type and are associated with a first host and a reference time period;
determine, based at least on the first group of host-based events, a first statistical deviation in the first activity type on the first host for the reference time period;
determine a second group of host-based events the indicate the same first activity type and are associated with the first host and a test time period;
determine, based at least on the second group of host-based events, a second statistical deviation in the first activity type on the first host for the test time period;
in response to determining that the first statistical deviation compared to the second statistical deviation satisfies a particular threshold, designate a host-based event from the second group as a seed;
generate a data item cluster based on the seed, wherein generating the data item cluster comprises:
adding the seed to the data item cluster; and
adding to the data item cluster one or more activity trend-related data items, from the activity trend-related data items, determined to be associated with the seed; and

determine scores for the data item cluster and a plurality of additional data items clusters generated based on host-based events; and

executing a workflow engine configured to at least:
cause presentation of the data item cluster and the plurality of additional data item clusters in a user interface of a client computing device; and
order the presented data item cluster and the plurality of additional data item clusters in the user interface based at least in part on the respective determined scores for the data item cluster and the plurality of additional data item clusters.

8. The computer-implemented method of claim 7 , wherein the activity trend-related data items include at least one of: data items associated with captured host-based events, Internet Protocol addresses, external domains, users, or computing devices, and wherein hosts comprise computing devices in a network.

9. The computer-implemented method of claim 7 further comprising:
by the one or more processors executing program instructions:
executing the cluster engine further configured to at least:
identify the one or more activity trend-related data items determined to be associated with the seed based at least on a clustering strategy, wherein the clustering strategy queries the host-based events and/or the activity trend-related data items to determine at least one of: the particular host associated with the seed, one or more host-based events associated with the particular host, one or more host-based events associated with the seed, users of the particular host, data items associated with the particular host, other hosts associated with the same particular activity type of host-based events, Internet Protocol addresses associated with the particular host, external domains associated with the seed, or computing devices associated with the particular host.

10. The computer-implemented method of claim 9 , wherein identifying one or more activity trend-related data items determined to be associated with the seed further comprises determining a particular

Assignees

• Palantir Technologies Inc

Inventors

• Singh Harkirat
• Weickert Brendan
• Sprague Matthew
• Kross Michael
• Borochoff Adam
• Menon Parvathy
• Harris Michael

Classifications

• G06Q40/03

  Credit; Loans; Processing thereof · CPC title

• G06Q40/00 Primary

  Finance; Insurance; Tax strategies; Processing of corporate or income taxes · CPC title

• G06F16/335

  Filtering based on additional data, e.g. user or group profiles (filtering in web context G06F16/9535, G06F16/9536) · CPC title

• G06F16/287

  Visualization; Browsing · CPC title

• G06F16/355

  Creation or modification of classes or clusters · CPC title