Query conversion for converting structured queries into unstructured queries for searching unstructured data

Patent metadata

Field	Value
Publication number	US-9934309-B2
Application number	US-201715473532-A
Country	US
Kind code	B2
Filing date	Mar 29, 2017
Priority date	Jul 31, 2013
Publication date	Apr 3, 2018
Grant date	Apr 3, 2018

Abstract

Technologies are described herein for executing queries expressed with reference to a structured query language against unstructured data. A user issues a structured query through a traditional structured data management (“SDM”) application. Upon receiving the structured query, an SDM driver analyzes the structured query and extracts a data structure from the unstructured data, if necessary. The structured query is then converted to an unstructured query based on the extracted data structure. The converted unstructured query may then be executed against the unstructured data. Results from the query are reorganized into structured data utilizing the extracted data structure and are then presented to the user through the SDM application.

First claim

What is claimed is:

1. A computer-implemented method comprising:
storing unstructured data in an unstructured data store, wherein at least some of the unstructured data remains unstructured in the unstructured data store;
receiving, at a query converter, a structured query in a structured query language from an application;
identifying a field in data stored in the unstructured data store, based on an extraction rule that specifies where to find a subportion of text within a segment of the data;
generating, by the query converter, a second query in a second query language associated with the unstructured data store, based on the structured query;
causing execution of the second query against the unstructured data stored in the unstructured data store;
receiving a result of execution of the second query against the unstructured data stored in the unstructured data store, the result including a value for the field; and
causing an indication of the result to be provided to the application for output to a user.

2. A computer-implemented method as recited in claim 1 , wherein at least some of the unstructured data is stored in the unstructured data store in a JSON format or a JSON-based format.

3. A computer-implemented method as recited in claim 1 , wherein the unstructured data store is not compatible with structured query language queries.

4. A computer-implemented method as recited in claim 1 , wherein at least some of the unstructured data is stored in the unstructured data store in a JSON format or a JSON-based format, and wherein the unstructured data store is not compatible with structured query language queries.

5. A computer-implemented method as recited in claim 1 , wherein the unstructured data store does not include a relational database.

6. A computer-implemented method as recited in claim 1 , wherein the unstructured data comprises time-series data, and the unstructured data store comprises a time-series data store.

7. A computer-implemented method as recited in claim 1 , wherein the unstructured data comprises time-series data, the unstructured data store comprises a time-series data store, and at least some of the unstructured data is stored in the unstructured data store in a JSON format or a JSON-based format.

8. A computer-implemented method as recited in claim 1 , further comprising:
in response to receiving the result of execution of the second query against the unstructured data stored in the unstructured data store, converting the result into a format compatible with an output capability of the application, prior to causing the indication of the result to be provided to the application for output to the user.

9. A computer-implemented method as recited in claim 1 , further comprising:
applying a schema to the data in the unstructured data store after storing data in the unstructured data store, to impose a structure on the unstructured data.

10. A computer-implemented method as recited in claim 1 , wherein the unstructured data includes data indicative of performance or operation of one or more components of an information technology environment.

11. A computer-implemented method as recited in claim 1 , wherein the unstructured data includes data indicative of performance or operation of one or more components of an information technology environment, including log data.

12. A computer-implemented method as recited in claim 1 , further comprising:
segmenting the unstructured data into a plurality of events; and
generating a timestamp for each event;
wherein the storing unstructured data in the unstructured data store comprises storing the events in the unstructured data store in association with associated timestamps.

13. A computer-implemented method as recited in claim 1 , further comprising:
transmitting a pilot query to the unstructured data store;
responsive to the pilot query, receiving a plurality of records generated by execution of the pilot query against the unstructured data in the unstructured data store, each record including one or more pairs of field names and values;
automatically identifying field names and data types from the pairs in the plurality of records generated by execution of the pilot query against the unstructured data store; and
defining a first set of fields that corresponds to the identified field names and data types.

14. A computer-implemented method as recited in claim 1 , further comprising:
transmitting a pilot query to the unstructured data store prior to the receiving the structured query at the query converter;
responsive to the pilot query, receiving a plurality of records generated by execution of the pilot query against the unstructured data in the unstructured data store, each record including one or more pairs of field names and values;
automatically identifying field names and data types from the pairs in the plurality of records generated by execution of the pilot query against the unstructured data store; and
defining a first set of fields that corresponds to the identified field names and data types.

15. A computer-implemented method as recited in claim 1 , further comprising:
transmitting a pilot query to the unstructured data store;
responsive to the pilot query, receiving a plurality of records generated by execution of the pilot query against the unstructured data in the unstructured data store, each record including one or more pairs of field names and values;
automatically identifying field names and data types from the pairs in the plurality of records generated by execution of the pilot query against the unstructured data store; and
defining a first set of fields that corresponds to the identified field names and data types;
wherein the causing execution of the second query against the unstructured data stored in the unstructured data store causes one or more values for one or more fields included in the second query to be extracted from the unstructured data store;
the method further comprising:
receiving the extracted values for the fields included in the unstructured query from the unstructured data store.

16. A computer-implemented method as recited in claim 1 , further comprising:
transmitting a pilot query to the unstructured data store;
responsive to the pilot query, receiving a plurality of records generated by execution of the pilot query against the unstructured data in the unstructured data store, each record including one or more pairs of field names and values;
automatically identifying field names and data types from the pairs in the plurality of records generated by execution of the pilot query against the unstructured data store;
defining a first set of fields that corresponds to the identified field names and data types; and
caching the first set of fields.

17. A computer-implemented method as recited in claim 1 , further comprising:
transmitting a pilot query to the unstructured data store on a subset of records in the unstructured data store, wherein the subset of the records is of a definable size;
responsive to the pilot query, receiving a plurality of records generated by execution of the pilot query against the unstructured data in the unstructured data store, each record including one or more pairs of field names and values;
automatically identifying field names and data types from the pairs in the plurality of records generated by execution of the pilot query against the unstructured data store, as a function of a formatting of the records; and
defining a first set of fields that corresponds to the identified field names and dat

Assignees

• Splunk Inc

Inventors

• Neeman Itay
• Lovering Bradford H

Classifications

• G06F17/30696

  Physics · mapped topic

• G06F17/3066 Primary

  Physics · mapped topic

• G06F16/80 Primary

  of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML (content-based retrieval of web data G06F16/95) · CPC title

• G06F16/338

  Presentation of query results · CPC title

• G06F16/9032

  Query formulation · CPC title