Methods and systems for performing transparent object migration across storage tiers

Patent metadata

Field	Value
Publication number	US-9824108-B2
Application number	US-201113089928-A
Country	US
Kind code	B2
Filing date	Apr 19, 2011
Priority date	Apr 19, 2010
Publication date	Nov 21, 2017
Grant date	Nov 21, 2017

Abstract

In accordance with embodiments, there are provided mechanisms and methods for performing transparent object migration across storage tiers. In an embodiment and by way of example, a method for appending data to large data volumes is provided. The method embodiment includes a) setting a CustomEntityOption bit that determines (at object creation time) where the object is stored, either in the relational or the non-relational data store portion, b) loading the CustomEntityOption bit in a cached CustomEntityDefinition, c) showing the CustomEntityOption bit as EntityInfo, and d) allowing custom object definition and Metadata API functionality when the bit is shown.

First claim

What is claimed is:

1. In a combined data repository having a relational data store and a non-relational data store, a method for updating search index metadata, wherein the combined data repository to store objects having an associated indicator set at object creation to indicate whether the objects are to be stored in the relational data store or in the non-relational data store, the combined data repository to:
retrieve a complete search index row from the non-relational data store, the search index row having a plurality of columns having row values containing metadata about the search index row;
update the metadata and at least a portion of the row values of the plurality of columns to form an updated search index row having the plurality of columns, wherein only a subset of the plurality of columns is updated;
send the updated search index row with all of its row values to the non-relational data store;
replace the search index row in the non-relational data store;
upsert updated column values from the updated search index row to a corresponding search index row in a table of the relational data store;
define the search index metadata by a cached custom index object;
send the metadata through requests to the non-relational data store as part of a specified object; and
force a resend of the search index metadata by modifying a sequence number for a corresponding table and owner combination upon index creation or deletion, wherein the resend is not forced unless the sequence number changes.

2. The method of claim 1 , wherein updating includes surfacing a SQL UPDATE statement that allows a subset of the columns to be updated.

3. The method of claim 1 , wherein the search index is maintained on a server side through a SyncTableCache object that is responsible for maintaining a snapshot of unflushed rows for each indexed data table of the relational data store to ensure that all operations are working off of up-to-date data.

4. The method of claim 3 , further comprising upserting the updated column values at the same time that the non-relational data store is updated.

5. The method of claim 1 , wherein deleted data rows cause a corresponding search index row to be deleted, inserted data rows cause a corresponding search index row to be inserted, updated data rows cause a corresponding old search index row to be deleted and a new index row to be inserted, and search index rows are flushed together with the data rows in a flush operation.

6. The method of claim 1 , further comprising:
calculating index statistics through normal queries of the non-relational data store,
storing the calculated index statistics in existing statistics tables, and
periodically updating the statistics using a scheduled batch process.

7. The method of claim 6 , wherein calculating the index statistics is performed when an index has been completely built.

8. The method of claim 6 , wherein calculating the index statistics is performed when an index is marked as active.

9. The method of claim 6 , wherein a normal query comprises SELECT count(Id) totalCount,count_distinct(<first indexed field>) distinctCount FROM <data table>.

10. The method of claim 6 , wherein a normal query comprises SELECT <first indexed field> value, COUNT(Id) totalCount FROM <data table> GROUP BY <first indexed field> HAVING COUNT(Id) >100.

11. The method of claim 1 , wherein updating the column values further comprises:
running an optimize routine in a new index creation mode for all existing data segments,
writing out new index segments during index creation of the optimize routine,
creating index rows for unflushed data rows,
enabling incremental index maintenance to cause the creation of index rows for any data that changes while the creation of the index row is in progress, and
marking the created index rows as active when the creation of index rows is completed to enable queries to begin using the created index rows.

12. A combined data repository being a data storage device and having a relational data store, a non-relational data store, and an application server for updating search index metadata, wherein objects stored in the combined data repository have an associated indicator set at object creation to indicate whether corresponding objects are to be stored in the relational data store or in the non-relational data store, the application server to perform operations comprising:
retrieving a complete search index row from the non-relational data store, the search index row having a plurality of columns having row values containing metadata about the search index row;
updating the metadata and at least a portion of the row values of the plurality of columns to form an updated search index row having the plurality of columns, wherein only a subset of the plurality of columns is updated;
sending the updated search index row with all of its row values to the non-relational data store;
replacing the search index row in the non-relational data store;
upserting updated column values from the updated search index row to a corresponding search index row in a table of the relational data store;
defining the search index metadata by a cached custom index object;
sending the metadata through requests to the non-relational data store as part of a specified object; and
forcing a resend of the search index metadata by modifying a sequence number for a corresponding table and owner combination upon index creation or deletion, wherein the resend is not forced unless the sequence number changes.

13. The combined data repository of claim 12 , wherein updating includes surfacing a SQL UPDATE statement that allows a subset of the columns to be updated.

14. The combined data repository of claim 12 , wherein the search index is maintained on a server side through a SyncTableCache object that is responsible for maintaining a snapshot of unflushed rows for each indexed data table of the relational data store to ensure that all operations are working off of up-to-date data.

15. The combined data repository of claim 14 , the application server further updating each indexed data table at the same time that the non-relational data store is updated.

16. The combined data repository of claim 12 , wherein deleted data rows cause a corresponding search index row to be deleted, inserted data rows cause a corresponding search index row to be inserted, updated data rows cause a corresponding old search index row to be deleted and a new search index row to be inserted, and search index rows are flushed together with the data rows in a flush operation.

17. The combined data repository of claim 12 , the application server further:
calculating index statistics through normal queries of the non-relational data store,
storing the calculated index statistics in existing statistics tables, and
periodically updating the statistics using a scheduled batch process.

18. The combined data repository of claim 12 , wherein updating the column values further comprises:
running an optimize routine in a new index creation mode for all existing data segments,
writing out new index segments during index creation of the optimize routine,
creating index rows for unflushed data rows,
enabling incremental index maintenance to cause the creation of index rows for data that changes while the creation of index row is in progress, and
marking the creation of index rows as active when the creation of index rows is complete to enable queries to begin using the created index rows.

19. In a combined data repo

Assignees

• Taylor James
• Eidson Bill C
• Oliver Kevin
• Salesforce Com Inc

Inventors

• Taylor James
• Eidson Bill C
• Oliver Kevin

Classifications

• G06F17/30336 Primary

  Physics · mapped topic

• G06F16/2272 Primary

  Management thereof · CPC title