On Tuning the Sorted Neighborhood Method for Record Comparisons in a Data Deduplication Pipeline
[ 1 ] Instytut Informatyki, Wydział Informatyki i Telekomunikacji, Politechnika Poznańska | [ P ] employee
2023
chapter in monograph / paper
english
- data deduplication pipeline
- customers’ records deduplication
- sorted neighborhood
- moving window size
EN Assuring high quality of data stored in information systems (ISs) is challenging and it is one of concerns of companies. Typically, data stored in ISs are not free from errors, which include among others wrong and missing values as well as duplicates. Data deduplication has received a lot of attention from the research community. The research efforts have resulted in a state-of-the-art data deduplication pipeline, supported by software tools and algorithms. One of the tasks in the pipeline consists in reducing the complexity of records comparisons. This task is known as blocking. Multiple algorithms for blocking have been proposed and one of them is the sorted neighborhood method. In this paper, we focus on tuning and evaluating the method on a real data set composed of 5.5M of customer records. To the best of our knowledge, this is the largest real data set being used in research. The findings reported in this paper come from a R &D project run for a big company in a financial sector.
18.08.2023
164 - 178
20
70