Multiple sensor session and log information compression and correlation system

Patent metadata

Field	Value
Publication number	US-10698927-B1
Application number	US-201715672069-A
Country	US
Kind code	B1
Filing date	Aug 8, 2017
Priority date	Aug 30, 2016
Publication date	Jun 30, 2020
Grant date	Jun 30, 2020

Abstract

Methods, systems and apparatus, including computer programs encoded on computer storage media for compressing sensor log information. One of the methods includes accessing log information maintained in one or more databases, the log information being generated in response to actions associated with entities, and the log information indicative of respective sessions for which one or more logs were generated, each log indicating an entity. Log information is grouped according to entity. One or more logs associated with respective sessions based on the grouped log information. Compressed logs are generated from logs associated with respective sessions based on compression rules.

First claim

What is claimed is:

1. A method comprising:
by a system comprising one or more processors,
accessing log information maintained in one or more databases, the log information being generated in response to actions associated with entities, the log information including, at least, information indicative of respective sessions for which one or more logs were generated, the actions being indicative of events associated with sessions, each log including a plurality of columns and indicating an entity in a first column of the plurality of columns;
grouping log information according to entity;
determining, for respective entities based on the grouped log information, one or more logs associated with respective sessions; and
generating, from logs associated with each session, a compressed log based on one or more compression rules, wherein generating the compressed log associated with the each session comprises:
identifying a plurality of actions associated with the each session, the plurality of actions being included in a plurality of rows of a second column of the logs associated with the each session, and
concatenating the plurality of actions into a single row of the plurality of rows of the second column, wherein remaining rows of the plurality of rows of the second column are discarded.

2. The method of claim 1 , wherein the log information is included in one or more database tables, and wherein the method further comprises:
updating the one or more database tables to include the compressed logs, thereby reducing a size associated with the database tables.

3. The method of claim 1 , wherein each log indicates a timestamp associated with its generation, and wherein grouping log information according to entity comprises:
grouping logs according to entity; and
for each entity, sorting log information associated with the entity according to timestamp.

4. The method of claim 1 , wherein a compression rule specifies a rule associated with combining, discarding, or selecting, information indicated in the logs associated a session.

5. The method of claim 1 , wherein the log information is included in one or more database tables, and wherein at least two logs associated with a particular session are separated in the database tables by hundreds or thousands of logs.

6. The method of claim 1 , further comprising:
receiving, via application programing interface (API) calls, requests for compressed logs, the requests being received over one or more networks by an outside system associated with the entities; and
providing, to the outside system, the compressed logs, such that the outside system can perform corrective actions in response to analyzing the compressed logs.

7. The method of claim 1 , further comprising:
accessing information indicating subscriptions to types of information by user devices;
identifying, based on the compressed logs, a particular compressed log that includes a type of information to which a particular user device subscribes; and
generating notification information for providing to the user device, the notification information automatically activating the particular user device to perform one or more corrective actions.

8. A system comprising one or more computers and computer storage media storing instructions that when executed by the one or more computers, cause the one or more computers to perform operations comprising:
accessing log information maintained in one or more databases, the log information being generated in response to actions associated with entities, the log information including, at least, information indicative of respective sessions for which one or more logs were generated the actions being indicative of events associated with sessions, each log including a plurality of columns and indicating an entity in a first column of the plurality of columns;
grouping log information according to entity;
determining, for respective entities based on the grouped log information, one or more logs associated with respective sessions; and
generating, from logs associated with each session, a compressed log based on one or more compression rules, wherein generating the compressed log associated with the each session comprises:
identifying a plurality of actions associated with the each session, the plurality of actions being included in a plurality of rows of a second column of the logs associated with the each session, and
concatenating the plurality of actions into a single row of the plurality of rows of the second column, wherein remaining rows of the plurality of rows of the second column are discarded.

9. The system of claim 8 , wherein the log information is included in one or more database tables, and wherein the operations further comprise:
updating the one or more database tables to include the compressed logs, thereby reducing a size associated with the database tables.

10. The system of claim 8 , wherein each log indicates a timestamp associated with its generation, and wherein grouping log information according to entity comprises:
grouping logs according to entity; and
for each entity, sorting log information associated with the entity according to timestamp.

11. The system of claim 8 , wherein a compression rule specifies a rule associated with combining, discarding, or selecting, information indicated in the logs associated a session.

12. The system of claim 8 , wherein the log information is included in one or more database tables, and wherein at least two logs associated with a particular session are separated in the database tables by hundreds or thousands of logs.

13. The system of claim 8 , wherein the operations further comprise:
receiving, via application programing interface (API) calls, requests for compressed logs, the requests being received over one or more networks by an outside system associated with the entities; and
providing, to the outside system, the compressed logs, such that the outside system can perform corrective actions in response to analyzing the compressed logs.

14. The system of claim 8 , wherein the operations further comprise:
accessing information indicating subscriptions to types of information by user devices;
identifying, based on the compressed logs, a particular compressed log that includes a type of information to which a particular user device subscribes; and
generating notification information for providing to the user device, the notification information automatically activating the particular user device to perform one or more corrective actions.

15. Non-transitory computer storage media storing instructions that when executed by a system of one or more computers cause the system to perform operations comprising:
accessing log information maintained in one or more databases, the log information being generated in response to actions associated with entities, the log information including, at least, information indicative of respective sessions for which one or more logs were generated, the actions being indicative of events associated with sessions, each log including a plurality of columns and indicating an entity in a first column of the plurality of columns;
grouping log information according to entity;
determining, for respective entities based on the grouped log information, one or more logs associated with respective sessions; and
generating, from logs associated with each session, a compressed log based on one or more compression rules, wherein generating the compressed log associated with the each session comprises:
identifying a plurality of actions associated with the each session, the

Assignees

• Palantir Technologies Inc

Inventors

• Chin Daniel
• Dries Daniel

Classifications

• G06F16/2358 Primary

  Change logging, detection, and notification (replication G06F16/27) · CPC title

• G06F16/285 Primary

  Clustering or classification · CPC title

• G06F16/2282

  Tablespace storage structures; Management thereof · CPC title

• G06F16/1744

  using compression, e.g. sparse files · CPC title

• G06F16/2322

  using timestamps · CPC title