Merging data from a source location into a target location

Patent metadata

Field	Value
Publication number	US-9864763-B2
Application number	US-201214398249-A
Country	US
Kind code	B2
Filing date	Jun 1, 2012
Priority date	Jun 1, 2012
Publication date	Jan 9, 2018
Grant date	Jan 9, 2018

Abstract

In an example, data from a source location is merged into a target location containing existing data, in which both the source location and the target location contain tuples of data. For each tuple in the source location, during a single operation, a determination is made as to whether there is a matched tuple in the target location that satisfies a predetermined condition. For each matched tuple that satisfies the predetermined condition, the matched tuple in the target location is updated with a count value that is equal to a count of the matched tuple in the source location and the target location. In addition, for each tuple that does not have a matched tuple that satisfies the predetermined condition, the unmatched tuple is inserted into the target location.

First claim

What is claimed is:

1. A method for merging data from a source location into a target location containing existing data, wherein both the source location and the target location contain tuples of data, the method comprising:
for each tuple in the source location, during a single operation,
determining, by a processor, whether there is a matched tuple in the target location that satisfies a predetermined condition;
for each matched tuple that satisfies the predetermined condition, updating the matched tuple in the target location with a count value that is equal to a count of the matched tuple in the source location and the target location, wherein the updating comprises:
inserting the matched tuple into a delete vector;
marking the matched tuple in the delete vector as removed from the target location, without physically removing the matched tuple from the target location;
inserting the updated matched tuple into the target location; and

for each tuple that does not have a matched tuple that satisfies the predetermined condition, inserting the unmatched tuple into the target location;
associating a first indicator with the matched tuple in the target location, the first indicator indicating that the matched tuple in the target location is an existing tuple; and
associating a second indicator with the unmatched tuple inserted into the target location, the second indicator indicating that the unmatched tuple is new in the target location.

2. The method according to claim 1 , further comprising:
scanning both the source location and the target location to determine the tuples contained in each of the source location and the target location;
wherein determining whether there is a matched tuple in the target location that satisfies the predetermined condition comprises right outer joining the source location and the target location to determine which tuples from the source location are to be inserted into the target location and which tuples in the target location are to be updated.

3. The method according to claim 2 , wherein right outer joining the source location and the target location comprises adding a matching Boolean (M) to each output tuple of the right outer join to mark whether the tuple is new (M=false) or existing (M=true), wherein the first indicator is M=true, and the second indicator is M=false;
moving matched tuples (M=true) into the delete vector; and
inserting unmatched tuples (M=false) and matched tuples (M=true) into the target location.

4. The method according to claim 3 , further comprising:
applying a first filter to output the unmatched tuples into the delete vector;
applying expressions for the matched and unmatched tuples; and
applying a second filter to output the unmatched tuples and the matched tuples prior to inserting the unmatched tuples and the matched tuples into the target location.

5. The method according to claim 4 , further comprising:
implementing a sideways information passing operation that eliminates unmatched tuples from the target location before performing the right outer joining; and
wherein right outer joining the source location and the target location further comprises right outer joining the source location and the target location following implementation of the sideways information passing operation.

6. The method according to claim 1 , wherein the target location comprises a projection, the method further comprising:
scanning both the source location and the projection to determine the tuples contained in each of the source location and the projection;
wherein determining whether there is a matched tuple in the target location that satisfies the predetermined condition comprises determining whether there is a matched tuple in the projection by right outer merge joining the source location and the projection to determine which tuples from the source location are to be inserted into the projection and which tuples in the projection are to be updated.

7. The method according to claim 6 , wherein the target location comprises a plurality of projections, the method further comprising:
selecting one of the plurality of projections to scan based upon at least one factor, wherein the scanning comprises scanning both the source location and the selected one of the plurality of projections.

8. The method according to claim 7 , further comprising:
scanning a dimension table containing descriptive attributes of events;
joining the descriptive attributes of events in the dimension table with data contained in an additional projection with the right outer merge joined source location and the selected one of the plurality of projections;
inserting the unmatched tuples from the right outer merge join into each of the plurality of projections; and
updating the matched tuples from the right outer merge join in each of the plurality of projections.

9. The method according to claim 8 , wherein the existing data of the target location is distributed across different nodes, the method further comprising:
determining that the distributed existing data is to be redistributed among the different nodes; and
redistributing the distributed existing data among the different nodes in response to a determination that the distributed existing data is to be redistributed among the different nodes.

10. An apparatus for merging data from a source location into a target location containing existing data, wherein both the source location and the target location contain tuples of data, and wherein the target location comprises a column-based database, the apparatus comprising:
a processor;
a non-transitory storage medium storing instructions executable on the processor to:
scan the source location and the target location to determine the tuples contained in each of the source location and the target location;
right outer join the source location and the target location to determine unmatched tuples from the source location to be inserted into the target location, and matched tuples in the target location to be updated;
for each of the matched tuples in the target location that are to be updated, update the matched tuple in the target location with a count value that is equal to a number of times the matched tuple appears in the source location and in the target location, the updating comprising:
inserting the matched tuple into a delete vector;
marking the matched tuple in the delete vector as removed from the target location, without physically removing the matched tuple from the target location:
inserting the updated matched tuple into the target location; and
associating a first indicator with the updated matched tuple in the target location, the first indicator indicating that the matched tuple in the target location is an existing tuple; and

for each of the unmatched tuples in the source location that are to be inserted into the target location,
insert the unmatched tuple into the target location; and
associate a second indicator with the unmatched tuple inserted into the target location, the second indicator indicating that the unmatched tuple is new in the target location.

11. The apparatus according to claim 10 , wherein the instructions are executable on the processor to:
add a matching Boolean (M) to each output tuple of the right outer join to mark whether the output tuple is new (M=false) or existing (M=true), wherein the first indicator is M=true, and the second indicator is M=false;
insert existing tuples (M=true) into the delete vector; and
insert new tuples (M=false) and existing tuples (M=true) into the target location.

12. The apparatus according to claim 11 , wherein the instruction

Assignees

• Tran Hoa Binh Nga
• Lamb Andrew Allinson
• Fuller Matthew Steven
• Entit Software Llc

Inventors

• Tran Hoa Binh Nga
• Lamb Andrew Allinson
• Fuller Matthew Steven

Classifications

• G06F17/30678

  Physics · mapped topic

• G06F17/30563

  Physics · mapped topic

• G06F17/30345

  Physics · mapped topic

• G06F17/30292 Primary

  Physics · mapped topic

• G06F16/23 Primary

  Updating · CPC title