Next generation near real-time indexing

Patent metadata

Field	Value
Publication number	US-11216430-B2
Application number	US-201715787283-A
Country	US
Kind code	B2
Filing date	Oct 18, 2017
Priority date	Dec 31, 2012
Publication date	Jan 4, 2022
Grant date	Jan 4, 2022

Abstract

Methods and systems to build and utilize a search infrastructure are described. The system generates, by a first indexing subsystem, full indexes of data and sends the full indexes to a plurality of query node servers via a distribution system. The system generates, mini-indexes associated with the full indexes and sends the mini-indexes to the plurality of query node servers. The system causes the plurality of query nodes to update indexing information based on the full indexes and the mini-indexes.

First claim

We claim:

1. A computer-implemented method comprising:
generating, by a first indexing subsystem, full indexes of data;
sending the full indexes to a plurality of query node servers via a distribution system for configuring the plurality of query node servers to perform search queries using the full indexes;
generating mini-indexes associated with the full indexes;
selecting, based on an update priority of the mini-indexes, to send the mini-indexes to the plurality of query node servers via a first path, a second path different from the first path, or both, wherein the first path includes the distribution system and the second path excludes the distribution system, wherein the update priority of the mini-indexes has a first priority level or a second priority level, wherein the first priority level exceeds the second priority level, and wherein the selecting comprises:
selecting to send the mini-indexes to the plurality of query node servers via at least the second path based at least in part on the update priority of the mini-indexes having the first priority level; or
selecting to send the mini-indexes to the plurality of query node servers via the first path based at least in part on the update priority of the mini-indexes having the second priority level; and

sending, based on the selecting, the mini-indexes to the plurality of query node servers via the first path, the second path, or both, for configuring the plurality of query node servers to update indexing information using the full indexes and the mini-indexes.

2. The method of claim 1 , wherein the first indexing subsystem represents a map-reduced indexing system and the mini-indexes associated with the full indexes are generated by a second indexing subsystem that is different than the first indexing subsystem.

3. The method of claim 2 , wherein the second indexing subsystem representing a daemon-based indexing system that monitors for and receives in near real-time event notifications, the event notifications based on a priority ordering of events by application servers.

4. The method of claim 3 , further comprising:
publishing the updated indexing information to one or more publishing channels in the distribution system based on a normal priority document update, and publishing the updated indexing information to the plurality of query node servers based on a higher priority document update.

5. The method of claim 4 , further comprising:
updating the indexing information associated with normal priority document updates associated with the full indexes and high priority document updates associated with the mini-indexes at the plurality of query node servers;
receiving, over a network, a search query from a client machine;
identifying search results based on the search query and the updated indexing information; and
communicating the search results, over the network, to the client machine.

6. The method of claim 5 , further comprising:
processing the event notifications of the document updates in a message queue by indexing daemons and a daemon coordinator.

7. An indexing system comprising:
a plurality of query node servers, each of the plurality of query node servers being configured to perform search queries;
a first indexing subsystem;
a second indexing subsystem;
one or more computer processors; and
one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the indexing system to perform operations comprising:
generating, by the first indexing subsystem, full indexes of data;
sending the full indexes to the plurality of query node servers via a distribution system;
generating mini-indexes associated with the full indexes;
selecting, based on an update priority of the mini-indexes, to send the mini-indexes to the plurality of query node servers via a first path, a second path different from the first path, or both, wherein the first path includes the distribution system and the second path excludes the distribution system, wherein the update priority of the mini-indexes has a first priority level or a second priority level, wherein the first priority level exceeds the second priority level, and wherein the selecting comprises:
selecting to send the mini-indexes to the plurality of query node servers via at least the second path based at least in part on the update priority of the mini-indexes having the first priority level; or
selecting to send the mini-indexes to the plurality of query node servers via the first path based at least in part on the update priority of the mini-indexes having the second priority level; and

sending, based on the selecting, the mini-indexes to the plurality of query node servers via the first path, the second path, or both, for configuring the plurality of query node servers to update indexing information based on the full indexes and the mini-indexes.

8. The indexing system of claim 7 , wherein the first indexing subsystem represents a map-reduced indexing system and the mini-indexes associated with the full indexes are generated by the second indexing subsystem that is different than the first indexing subsystem.

9. The indexing system of claim 8 , wherein the second indexing subsystem represents a daemon-based indexing system that monitors for and receives in near real-time event notifications, the event notifications based on a priority ordering of events by application servers.

10. The indexing system of claim 9 , the operations further comprising:
publishing the updated indexing information to one or more publishing channels in the distribution system based on a normal priority document update, and publishing the updated indexing information to the plurality of query node servers based on a higher priority document update.

11. The indexing system of claim 10 , the operations further comprising:
updating the indexing information associated with normal priority document updates associated with the full indexes and high priority document updates associated with the mini-indexes at the plurality of query node servers;
receiving, over a network, a search query from a client machine;
identifying search results based on the search query and the updated indexing information; and
communicating the search results, over the network, to the client machine.

12. The indexing system of claim 11 , the operations further comprising:
processing the event notifications of the document updates in a message queue by indexing daemons and a daemon coordinator.

13. A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of an indexing system, cause the indexing system to perform operations comprising:
generating, by a first indexing subsystem of the indexing system, full indexes of data;
sending the full indexes to a plurality of query node servers via a distribution system for configuring the plurality of query node servers to perform search queries using the full indexes;
generating mini-indexes associated with the full indexes;
selecting, based on an update priority of the mini-indexes, to send the mini-indexes to the plurality of query node servers via a first path, a second path different from the first path, or both, wherein the first path includes the distribution system and the second path excludes the distribution system, wherein the update priority of the mini-indexes has a first priority level or a second priority level, wherein the first priority level exceeds the second priority level, and wherein the selecting comprises:
selecting to send the mini-indexes to the plurality of query node servers

Assignees

• Ebay Inc

Inventors

• Agarwal Swati
• Basu Amit
• Pan Thomas
• Sankaranthi Pradeep
• Whyte Nicholas

Classifications

• G06F16/2228 Primary

  Indexing structures · CPC title