Scaling available storage based on counting generated events

Patent metadata

Field	Value
Publication number	US-2016306871-A1
Application number	US-201514701301-A
Country	US
Kind code	A1
Filing date	Apr 30, 2015
Priority date	Apr 20, 2015
Publication date	Oct 20, 2016
Grant date	—

Abstract

A data intake and query system measures an amount of raw data ingested by the system during defined periods of time. As used herein, ingesting raw data generally refers to receiving the raw data from one or more computing devices and processing the data for storage and searchability. Processing the data may include, for example, parsing the raw data into “events,” where each event includes a portion of the received data and is associated with a timestamp. Based on a calculated number of events generated by the system during one or more defined time periods, the system may calculate various metrics including, but not limited to, a number of events generated during a particular day, a number of events generated per day over a period of time, a maximum number of events generated in a day over a period of time, an average number of events generated per day, etc.

First claim

What is claimed is:

1 . A method comprising:
receiving raw data from one or more devices; generating a plurality events from the raw data by:
parsing the raw data into a plurality of events, each event of the plurality of events including a portion of the raw data;
determining a respective timestamp for each event of the plurality of events;
determining a number of events of the plurality of events that were generated during a defined time period; comparing the number of events that were generated during the defined time period to an allocated event count; in response to a determination that the number of events that were generated during the defined time period has reached an allocated event count, performing one or more actions to raw data received subsequent to the allocated event count being reached.

2 . The method of claim 1 , wherein the allocated event count specifies a maximum number of events generated during the defined time period.

3 . The method of claim 1 , wherein the allocated event count specifies a maximum average number of events generated during the defined time period.

4 . The method of claim 1 , wherein performing one or more actions to raw data received subsequent to the allocated event count being reached includes storing events generated from the raw data in a non-searchable index.

5 . The method of claim 1 , wherein performing one or more actions to raw data received subsequent to the allocated event count being reached includes storing events generated from the raw data in a non-searchable index;
enabling access to the events that are stored in the non-searchable index when an increase to the allocated event count is purchased.

6 . The method of claim 1 , wherein performing one or more actions to raw data received subsequent to the allocated event count being reached includes storing events generated from the raw data in a non-accessible index.

7 . The method of claim 1 , wherein performing one or more actions to raw data received subsequent to the allocated event count being reached includes automatically deleting events that exceed the allocated event count.

8 . The method of claim 1 , further comprising, in response to the determination that the number of events that were generated during the defined time period has reached the allocated event count, automatically increasing the allocated event count.

9 . The method of claim 1 , further comprising storing at least one event of the plurality of events in an index.

10 . The method of claim 1 , further comprising:
storing at least one event of the plurality of events in an index; in response to the determination that the number of events that were generated during the defined time period has reached the allocated event count, ceasing to generate new events based on the raw data received from the one or more devices.

11 . The method of claim 1 , further comprising, in response to the determination that the number of events that were generated during the defined time period has reached the allocated event count, ceasing to accept raw data from the one or more devices.

12 . The method of claim 1 , further comprising, in response to the determination that the number of events that were generated during the defined time period has reached the allocated event count, generating an alert indicating that the allocated event count has been reached.

13 . The method of claim 1 , further comprising, in response to the determination that the number of events that were generated during the defined time period has reached the allocated event count, sending an alert message to a particular user.

14 . The method of claim 1 , wherein determining the number of events of the plurality of events that were generated during the defined time period includes determining that the number of events are associated with a particular user account of a plurality of user accounts.

15 . The method of claim 1 , wherein determining the number of events of the plurality of events that were generated during the defined time period includes determining that the number of events are associated with a particular project of a plurality of projects.

16 . The method of claim 1 , wherein parsing the raw data into a plurality of events further comprises determining event boundaries for the plurality of events.

17 . The method of claim 1 , wherein the plurality of events are searchable using a late-binding schema comprising one or more extraction rules for extracting values from the events.

18 . The method of claim 1 , wherein the defined time period corresponds to one or more days.

19 . The method of claim 1 , wherein the defined time period corresponds to one or more seconds.

20 . The method of claim 1 , further comprising calculating an average number of events that were generated over a plurality of time periods.

21 . The method of claim 1 , further comprising calculating a fee amount based on the number of events of the plurality of events that were generated during the defined time period.

22 . The method of claim 1 , wherein the one or more devices are managed by a managed security service provider (MSSP).

23 . The method of claim 1 , further comprising calculating a number of events that are stored in one or more particular indexes of a plurality of indexes.

24 . The method of claim 1 , wherein the raw data is associated with a particular project of a plurality of projects, each project of the plurality of projects having an associated licensed amount of data ingestion.

25 . The method of claim 1 , further comprising:
in response to the determination that the number of events that were generated during the defined time period has reached the allocated event count, further determining whether a user account associated with the one or more devices is permitted to exceed the allocated event count; in response to determining that the user account is permitted to exceed the allocated event count, generating up to a threshold number of additional events.

26 . The method of claim 1 , further comprising:
in response to the determination that the number of events that were generated during the defined time period has reached the allocated event count, further determining whether a user account associated with the one or more devices is permitted to exceed the allocated event count; in response to determining that the user account is permitted to exceed the allocated event count:
generating up to a threshold number of additional events; and
charging a fee to the user account for exceeding the allocated event count.

27 . The method of claim 1 , further comprising:
wherein the one or more devices includes both a first set of devices associated with a first company and a second set of devices associated with a second company, each of the first set of devices and the second set of devices managed by a managed security service provider (MSSP); wherein the raw data includes first raw data received from the first set of devices and second raw data received from the second set of devices; wherein determining the number of events of the plurality of events that were generated during a defined time period includes determining a first number of events generated based on the first raw data and a second number of events generated based on the second raw data; causing display of a user in

Assignees

• Splunk Inc

Inventors

• Chauhan Vijay
• Shahbaz Banipal
• Hazekamp David

Classifications

• G06F16/22

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F16/2477

  Temporal data queries · CPC title

• G06F16/287 Primary

  Visualization; Browsing · CPC title

• G06F17/30601 Primary

  Physics · mapped topic

• G06F17/30312

  Physics · mapped topic