{"id":2208,"date":"2014-01-06T11:01:00","date_gmt":"2014-01-06T11:01:00","guid":{"rendered":"https:\/\/www.dab-europe.com\/\/articles\/identifying-master-data-duplicates-simple-isnt-it\/"},"modified":"2025-03-11T12:28:43","modified_gmt":"2025-03-11T12:28:43","slug":"identifying-master-data-duplicates-simple-isnt-it","status":"publish","type":"articles","link":"https:\/\/www.dab-europe.com\/en\/articles\/identifying-master-data-duplicates-simple-isnt-it\/","title":{"rendered":"Identifying master data duplicates \u2013 Simple, isn\u2019t it?"},"content":{"rendered":"

Identifying master data duplicates \u2013 Simple, isn\u2019t it?<\/h2>\n

Today we deal with a simple request as identifying master data duplicates. Many of us know the problem with duplicates: actually a customer or vendor should exist only once in a system, in fact there are multiple records, as duplicates were created from data migrations or merging systems. Therefore \u201cDupli, Cate\u201d can exist more than once.<\/p>\n

\n\n\n\n\n\n\n\n\n\n
Vendor No.<\/td>\nName<\/td>\nAddress<\/td>\nTaxID<\/td>\n<\/tr>\n
1001<\/td>\nDupli, Cate<\/td>\nAbbey Road 17, 94469 Langley<\/td>\nCC00000001<\/td>\n<\/tr>\n
5311<\/td>\nDupli, Cate<\/td>\nAbbey Rd.17, 94469 Langley<\/td>\nAB11223344<\/td>\n<\/tr>\n
6442<\/td>\nJohnny Walker<\/td>\nKingstr. 13, 55117 Oxford<\/td>\nED98-76-54-32<\/td>\n<\/tr>\n
9876<\/td>\nJohnnie Walker<\/td>\nKingstr. 13, 55117 Oxford<\/td>\nED98765432 <\/td>\n<\/tr>\n
9877<\/td>\nJohn Doe<\/td>\nForest Lane1, 77788 Hoboken<\/td>\nAB11223344<\/td>\n<\/tr>\n
9992<\/td>\nCharles Little<\/td>\nAbbey Rd.17, 94469 Langley<\/td>\nZZ55667799 <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

 <\/p>\n

Different types of spelling can be a reason: \u201eJohnny Walker\u201c already exists in the system, the user is searching for \u201eJohnnie Walker\u201c, but will not get a match and consequently enters a second master data record. <\/p>\n

Complex problems can result from that: the wrong vendor gets paid (or a reminder is posted to the wrong vendor, which may even lead to a duplicate payment). More errors occur in the processes, or slow down actions. <\/p>\n

How can such duplicates be identified? We want to show some aspects of the approach:<\/p>\n

    \n
  1. Choice of suitable search criteria<\/li>\n
  2. Selective vs. blurred search<\/li>\n
  3. Data preparation \/ cleansing<\/li>\n
  4. Problems comparing results<\/li>\n<\/ol>\n

    Suitable search criteria<\/h2>\n

    Obviously it would have no effect to compare names, as sign comparing search would find \u201eDupli, Cate\u201c, but not \u201eJohnny Walker\u201c , as it is different from \u201cJohnnie Walker\u201c. <\/p>\n

    \n\n\n\n\n\n\n\n\n\n
    Vendor No.
    <\/b><\/td>\n    		Name
    <\/b><\/td>\n    		Address
    <\/b><\/td>\n    		TaxID
    <\/b><\/td>\n<\/tr>\n
    1001<\/td>\nDupli, Cate<\/span><\/td>\nAbbey Road 17, 94469 Langley<\/td>\nCC00000001<\/td>\n<\/tr>\n
    5311<\/td>\nDupli, Cate <\/span><\/td>\nAbbey Rd.17, 94469 Langley<\/td>\nAB11223344<\/td>\n<\/tr>\n
    6442<\/td>\nJohnny Walker<\/td>\nKingstr. 13, 55117 Oxford<\/td>\nED98-76-54-32<\/td>\n<\/tr>\n
    9876<\/td>\nJohnnie Walker<\/td>\nKingstr. 13, 55117 Oxford<\/td>\nED98765432 <\/td>\n<\/tr>\n
    9877<\/td>\nJohn Doe<\/td>\nForest Lane1, 77788 Hoboken<\/td>\nAB11223344<\/td>\n<\/tr>\n
    9992<\/td>\nCharles Little<\/td>\nAbbey Rd.17, 94469 Langley<\/td>\nZZ55667799 <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

     <\/p>\n

    Here we need different approach, e.g. duplicate search on address, tax or phone numbers or banking details. It is important to use search combinations creatively. The more unique search criteria are used, the less \u201efalse positives\u201c (very low potential duplicates) you get. <\/p>\n

    Selective vs. blurred search<\/h2>\n

    Sometimes it is necessary to do a blurred search in order to find master data, which exists multiple times, that cannot be identified by conventional analysis, e.g. on sound or by looking at the Levenshtein distance. Modern analysis software provides these search features and therefore is able to find special types of duplicates. More matches however can contain more \u201efalse positives\u201c, which means more effort to be looked at in detail. <\/p>\n

    \n\n\n\n\n\n\n\n\n\n\n
    Vendor No.<\/th>\nName<\/th>\nAddress<\/th>\nTaxID<\/th>\n<\/tr>\n<\/thead>\n
    1001<\/td>\nDupli, Cate<\/span><\/td>\nAbbey Road 17, 94469 Langley<\/td>\nCC00000001<\/td>\n<\/tr>\n
    5311<\/td>\nDupli, Cate <\/span><\/td>\nAbbey Rd.17, 94469 Langley<\/td>\nAB11223344<\/td>\n<\/tr>\n
    6442<\/td>\nJohnny Walker<\/td>\nKingstr. 13, 55117 Oxford<\/td>\nED98-76-54-32<\/td>\n<\/tr>\n
    9876<\/td>\nJohnnie Walker<\/td>\nKingstr. 13, 55117 Oxford<\/td>\nED98765432 <\/td>\n<\/tr>\n
    9877<\/td>\nJohn Doe<\/td>\nForest Lane1, 77788 Hoboken<\/td>\nAB11223344<\/td>\n<\/tr>\n
    9992<\/td>\nCharles Little<\/td>\nAbbey Rd.17, 94469 Langley<\/td>\nZZ55667799 <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    Data preparation \/ cleansing<\/h2>\n\n\n\n\n\n\n\n\n\n
    Vendor
    <\/b><\/td>\n    		Name
    <\/b><\/td>\n    		Address
    <\/b><\/td>\n    		TaxID
    <\/b><\/td>\n    		TaxID clean
    <\/b><\/td>\n<\/tr>\n
    1001<\/td>\nDupli, Cate<\/td>\nAbbey Road 17, 94469 Langley<\/td>\nCC00000001<\/td>\nCC00000001<\/td>\n<\/tr>\n
    5311<\/td>\nDupli, Cate <\/td>\nAbbey Rd.17, 94469 Langley<\/td>\nAB11223344<\/td>\nAB11223344<\/td>\n<\/tr>\n
    6442<\/td>\nJohnny Walker<\/td>\nKingstr. 13, 55117 Oxford<\/td>\nED98-76-54-32<\/td>\nED98765432<\/td>\n<\/tr>\n
    9876<\/td>\nJohnnie Walker<\/td>\nKingstr. 13, 55117 Oxfordccc<\/td>\nED98765432 <\/td>\nED98765432<\/td>\n<\/tr>\n
    9877<\/td>\nJohn Doe<\/td>\nForest Lane1, 77788 Hoboken<\/td>\nAB11223344<\/td>\nAB11223344<\/td>\n<\/tr>\n
    9992<\/td>\nCharles Little<\/td>\nAbbey Rd.17, 94469 Langley<\/td>\nZZ55667799 <\/td>\nZZ55667799<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    Problems comparing results<\/h2>\n

    Running analysis with different criteria it will be difficult to join or compare the results. <\/p>\n

    A search based on \u201cName\u201d will find one duplicate (1001 \/ 5311), based on \u201eAddress\u201c two duplicates (5311 \/ 9992 und 6442 \/ 9876). Checking \u201cTaxID\u201d one duplicate is the result. Indirectly 9877 is related to 1001, although they do not share any direct criterion such as Name, Address, TaxID.  <\/p>\n

    \n\n\n\n\n\n\n\n\n\n
    Vendor No.
    <\/b><\/td>\n    		Name
    <\/b><\/td>\n    		Address
    <\/b><\/td>\n    		TaxID<\/b><\/td>\n<\/tr>\n
    1001<\/td>\nDupli, Cate<\/td>\nAbbey Road 17, 94469 Langley<\/td>\nCC00000001<\/td>\n<\/tr>\n
    5311<\/td>\nDupli, Cate<\/td>\nAbbey Rd.17, 94469 Langley<\/td>\nAB11223344<\/span><\/td>\n<\/tr>\n
    6442<\/td>\nJohnny Walker<\/td>\nKingstr. 13, 55117 Oxford<\/span><\/td>\nED98-76-54-32<\/td>\n<\/tr>\n
    9876<\/td>\nJohnnie Walker<\/td>\nKingstr. 13, 55117 Oxford<\/td>\nED98765432<\/td>\n<\/tr>\n
    9877<\/td>\nJohn Doe<\/td>\nForest Lane1, 77788 Hoboken<\/td>\nAB11223344<\/span><\/td>\n<\/tr>\n
    9992<\/td>\nCharles Little<\/td>\nAbbey Rd.17, 94469 Langley<\/td>\nZZ55667799<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

     <\/p>\n

    In summary we can say that a trivial request to "identify duplicates" on closer inspection may represent a challenge. Important in the planning of such projects are therefore a stable concept with a clear roadmap in terms of methodology, as well as a plan for the subsequent correction of the duplicates in the system. <\/p>\n","protected":false},"excerpt":{"rendered":"

    Identifying master data duplicates \u2013 Simple, isn\u2019t it? Today we deal with a simple request as identifying master data duplicates. Many of us know the problem with duplicates: actually a customer or vendor should exist only once in a system, in fact there are multiple records, as duplicates were created from data migrations or merging […]<\/p>\n","protected":false},"author":2,"featured_media":4684,"template":"","articles_category":[],"articles_tag":[91,92],"class_list":["post-2208","articles","type-articles","status-publish","has-post-thumbnail","hentry","articles_tag-data-analytics","articles_tag-sap"],"acf":[],"yoast_head":"\nIdentifying master data duplicates in the system<\/title>\n<meta name=\"description\" content=\"Efficient identification of master data duplicates \u2713 Suitable search criteria \u2713 Data cleansing \u2713 Exact & fuzzy search approaches \u2013 Read more here!\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dab-europe.com\/en\/articles\/identifying-master-data-duplicates-simple-isnt-it\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Identifying master data duplicates in the system\" \/>\n<meta property=\"og:description\" content=\"Efficient identification of master data duplicates \u2713 Suitable search criteria \u2713 Data cleansing \u2713 Exact & fuzzy search approaches \u2013 Read more here!\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dab-europe.com\/en\/articles\/identifying-master-data-duplicates-simple-isnt-it\/\" \/>\n<meta property=\"og:site_name\" content=\"dab\" \/>\n<meta property=\"article:modified_time\" content=\"2025-03-11T12:28:43+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.dab-europe.com\/wp-content\/uploads\/2024\/06\/G19A7525-1024x683.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"683\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/articles\\\/identifying-master-data-duplicates-simple-isnt-it\\\/\",\"url\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/articles\\\/identifying-master-data-duplicates-simple-isnt-it\\\/\",\"name\":\"Identifying master data duplicates in the system\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/articles\\\/identifying-master-data-duplicates-simple-isnt-it\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/articles\\\/identifying-master-data-duplicates-simple-isnt-it\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.dab-europe.com\\\/wp-content\\\/uploads\\\/2024\\\/06\\\/G19A7525-scaled.jpg\",\"datePublished\":\"2014-01-06T11:01:00+00:00\",\"dateModified\":\"2025-03-11T12:28:43+00:00\",\"description\":\"Efficient identification of master data duplicates \u2713 Suitable search criteria \u2713 Data cleansing \u2713 Exact & fuzzy search approaches \u2013 Read more here!\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/articles\\\/identifying-master-data-duplicates-simple-isnt-it\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/articles\\\/identifying-master-data-duplicates-simple-isnt-it\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/articles\\\/identifying-master-data-duplicates-simple-isnt-it\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.dab-europe.com\\\/wp-content\\\/uploads\\\/2024\\\/06\\\/G19A7525-scaled.jpg\",\"contentUrl\":\"https:\\\/\\\/www.dab-europe.com\\\/wp-content\\\/uploads\\\/2024\\\/06\\\/G19A7525-scaled.jpg\",\"width\":2560,\"height\":1707},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/articles\\\/identifying-master-data-duplicates-simple-isnt-it\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Articles\",\"item\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/articles\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Identifying master data duplicates \u2013 Simple, isn\u2019t it?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/#website\",\"url\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/\",\"name\":\"dab\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/#organization\",\"name\":\"dab\",\"url\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.dab-europe.com\\\/wp-content\\\/uploads\\\/2025\\\/02\\\/dab_logo-main-color-scaled.png\",\"contentUrl\":\"https:\\\/\\\/www.dab-europe.com\\\/wp-content\\\/uploads\\\/2025\\\/02\\\/dab_logo-main-color-scaled.png\",\"width\":2560,\"height\":1463,\"caption\":\"dab\"},\"image\":{\"@id\":\"https:\\\/\\\/www.dab-europe.com\\\/en\\\/#\\\/schema\\\/logo\\\/image\\\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Identifying master data duplicates in the system","description":"Efficient identification of master data duplicates \u2713 Suitable search criteria \u2713 Data cleansing \u2713 Exact & fuzzy search approaches \u2013 Read more here!","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dab-europe.com\/en\/articles\/identifying-master-data-duplicates-simple-isnt-it\/","og_locale":"en_US","og_type":"article","og_title":"Identifying master data duplicates in the system","og_description":"Efficient identification of master data duplicates \u2713 Suitable search criteria \u2713 Data cleansing \u2713 Exact & fuzzy search approaches \u2013 Read more here!","og_url":"https:\/\/www.dab-europe.com\/en\/articles\/identifying-master-data-duplicates-simple-isnt-it\/","og_site_name":"dab","article_modified_time":"2025-03-11T12:28:43+00:00","og_image":[{"width":1024,"height":683,"url":"https:\/\/www.dab-europe.com\/wp-content\/uploads\/2024\/06\/G19A7525-1024x683.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.dab-europe.com\/en\/articles\/identifying-master-data-duplicates-simple-isnt-it\/","url":"https:\/\/www.dab-europe.com\/en\/articles\/identifying-master-data-duplicates-simple-isnt-it\/","name":"Identifying master data duplicates in the system","isPartOf":{"@id":"https:\/\/www.dab-europe.com\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.dab-europe.com\/en\/articles\/identifying-master-data-duplicates-simple-isnt-it\/#primaryimage"},"image":{"@id":"https:\/\/www.dab-europe.com\/en\/articles\/identifying-master-data-duplicates-simple-isnt-it\/#primaryimage"},"thumbnailUrl":"https:\/\/www.dab-europe.com\/wp-content\/uploads\/2024\/06\/G19A7525-scaled.jpg","datePublished":"2014-01-06T11:01:00+00:00","dateModified":"2025-03-11T12:28:43+00:00","description":"Efficient identification of master data duplicates \u2713 Suitable search criteria \u2713 Data cleansing \u2713 Exact & fuzzy search approaches \u2013 Read more here!","breadcrumb":{"@id":"https:\/\/www.dab-europe.com\/en\/articles\/identifying-master-data-duplicates-simple-isnt-it\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dab-europe.com\/en\/articles\/identifying-master-data-duplicates-simple-isnt-it\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.dab-europe.com\/en\/articles\/identifying-master-data-duplicates-simple-isnt-it\/#primaryimage","url":"https:\/\/www.dab-europe.com\/wp-content\/uploads\/2024\/06\/G19A7525-scaled.jpg","contentUrl":"https:\/\/www.dab-europe.com\/wp-content\/uploads\/2024\/06\/G19A7525-scaled.jpg","width":2560,"height":1707},{"@type":"BreadcrumbList","@id":"https:\/\/www.dab-europe.com\/en\/articles\/identifying-master-data-duplicates-simple-isnt-it\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.dab-europe.com\/en\/"},{"@type":"ListItem","position":2,"name":"Articles","item":"https:\/\/www.dab-europe.com\/en\/articles\/"},{"@type":"ListItem","position":3,"name":"Identifying master data duplicates \u2013 Simple, isn\u2019t it?"}]},{"@type":"WebSite","@id":"https:\/\/www.dab-europe.com\/en\/#website","url":"https:\/\/www.dab-europe.com\/en\/","name":"dab","description":"","publisher":{"@id":"https:\/\/www.dab-europe.com\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dab-europe.com\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.dab-europe.com\/en\/#organization","name":"dab","url":"https:\/\/www.dab-europe.com\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.dab-europe.com\/en\/#\/schema\/logo\/image\/","url":"https:\/\/www.dab-europe.com\/wp-content\/uploads\/2025\/02\/dab_logo-main-color-scaled.png","contentUrl":"https:\/\/www.dab-europe.com\/wp-content\/uploads\/2025\/02\/dab_logo-main-color-scaled.png","width":2560,"height":1463,"caption":"dab"},"image":{"@id":"https:\/\/www.dab-europe.com\/en\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/www.dab-europe.com\/en\/wp-json\/wp\/v2\/articles\/2208","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dab-europe.com\/en\/wp-json\/wp\/v2\/articles"}],"about":[{"href":"https:\/\/www.dab-europe.com\/en\/wp-json\/wp\/v2\/types\/articles"}],"author":[{"embeddable":true,"href":"https:\/\/www.dab-europe.com\/en\/wp-json\/wp\/v2\/users\/2"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.dab-europe.com\/en\/wp-json\/wp\/v2\/media\/4684"}],"wp:attachment":[{"href":"https:\/\/www.dab-europe.com\/en\/wp-json\/wp\/v2\/media?parent=2208"}],"wp:term":[{"taxonomy":"articles_category","embeddable":true,"href":"https:\/\/www.dab-europe.com\/en\/wp-json\/wp\/v2\/articles_category?post=2208"},{"taxonomy":"articles_tag","embeddable":true,"href":"https:\/\/www.dab-europe.com\/en\/wp-json\/wp\/v2\/articles_tag?post=2208"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}