Next generation near real-time indexing

Patent metadata

Field	Value
Publication number	US-9805078-B2
Application number	US-201314145414-A
Country	US
Kind code	B2
Filing date	Dec 31, 2013
Priority date	Dec 31, 2012
Publication date	Oct 31, 2017
Grant date	Oct 31, 2017

Abstract

Methods and systems to build and utilize a search infrastructure are described. The system generates index information, including document updates and indexes. The system receives event notifications as the document updates are received and accumulates the index information until published. A query engine receives a search query from a client machine and identifies search results based on the query and the index information. The system communicates the search results, over the network, to the client machine.

First claim

We claim:

1. A computer system to rebuild indexing information in near real time, the computer system comprising:
processors;
a plurality of query node servers;
a non-transitory first indexing subsystem when executing on at least one processor among the processors cause the first indexing subsystem to generate full indexes to publish document updates by the plurality of query node servers via a distribution system, the first indexing system representing a map-reduced indexing system;
a non-transitory second indexing subsystem when executing on at least one processor among the processors cause the second indexing subsystem to generate mini-indexes associated with the full indexes to publish document updates by the plurality of query node servers indirectly via the distribution system or directly to the plurality of query node servers, the second indexing subsystem representing a daemon-based indexing system that monitors for and receives in near real-time event notifications and generates the mini-indexes in near real-time based on receiving the event notifications, the event notifications based on a priority ordering of events by application servers;
a non-transitory query engine when executing on at least one processor among the processors cause the computer system to:
update the indexing information at the plurality of query node servers based on the mini-indexes and the full indexes; and
publish the updated indexing information to one or more publishing channels in the distribution system based on a normal priority document updates, and publish the updated indexing information to the query nodes servers based on a higher priority document update.

2. The system of claim 1 , wherein the second indexing subsystem further comprises a message queuing subsystem having a message queue for the event notifications of the document updates published by application servers, the message queuing subsystem subscribed to by the second indexing subsystem.

3. The system of claim 2 , wherein the second indexing subsystem further comprises a daemon coordinator to coordinate the processing of the event notifications of the document updates in the message queue by indexing daemons.

4. The system of claim 1 , wherein the second indexing subsystem further comprises a process document module for retrieving at least a portion of the index information from a database and an accumulator module for generating accumulated index information to be consumed and served by the plurality of query node servers.

5. The system of claim 4 , wherein the database is a Hadoop database.

6. A computer implemented method for rebuilding indexing information in near real time, the method comprising:
generating, by a first indexing subsystem, full indexes to sending full indexes by the first indexing system by a plurality of query node servers via a distribution system, the first indexing system representing a map-reduced indexing system;
generating, by a second indexing subsystem, mini-indexes associated with the full indexes;
sending the mini-indexes, by the second indexing system, indirectly via the distribution system or directly to the plurality of query node servers to publish the document updates by the plurality of query node servers, the second indexing subsystem representing a daemon-based indexing system that monitors for and receives in near real-time event notifications, the event notifications based on a priority ordering of events by application servers;
updating the indexing information at the plurality of query node servers based on the mini-indexes and the full indexes; and
publishing the updated indexing information to one or more publishing channels in the distribution system based on a normal priority document update, and publish the updated indexing information to the query nodes servers based on a higher priority document update.

7. The method of claim 6 , further comprising:
updating index information associated with normal priority document updates associated with the full-indexes and high priority document updates associated with the mini-indexes at the plurality of query node servers;
receiving a search query, over a network, from a client machine and identifying search results based on the query and the updated index information; and
communicating the search results, over the network, to the client machine.

8. The method of claim 6 , further comprising processing the event notifications of the document updates in a message queue by indexing daemons and a daemon coordinator.

9. The method of claim 8 , further comprising receiving the event notifications of the document updates as re-ordered based on the priority ordering of events by the application servers.

10. The method of claim 6 , wherein retrieving the index information further comprises retrieving the index information from a Hadoop database.

11. The system of claim 1 , wherein the mini-indexes includes information identifying which of the mini-indexes is paired with which of the full indexes.

12. The system of claim 1 , wherein the application servers prioritizes business event streams based on event types.

13. The system of claim 1 , wherein the application servers generates a dual stream of output data, the dual stream including event notifications received by the second indexing system and actual data included in the events received by the first indexing system.

14. The system of claim 1 , wherein the application servers processes events that has been re-ordered based on a priority scheme.

15. The system of claim 1 , wherein the second indexing system delivers the mini-indexes to one or more channels.

16. The system of claim 1 , wherein the first indexing system generates the full indexes at a fixed schedule for the full indexes and the second indexing system generates the mini-indexes on-demand or at a fixed schedule for the mini-indexes.

Assignees

• Ebay Inc
• Ebay Inc

Inventors

• Agarwal Swati
• Basu Amit
• Pan Thomas
• Sankaranthi Pradeep
• Whyte Nicholas

Classifications

• G06F17/30321 Primary

  Physics · mapped topic

• G06F16/2228 Primary

  Indexing structures · CPC title