Duplicate Handling Mask Option

Enable the DuplicateHandlingMask option to remove valid duplicate components from the input address and enhance the address cleansing accuracy.

The DuplicateHandlingMask option allows the address cleanser to handle some of the duplicate information that is encountered during user input. Depending on the value set for the option, the duplicates present in the address can be removed either from the same line or across all levels of input.

Example of Address with Duplicate Data - without the DuplicateHandlingMask Option Enabled

[
 {
  "pcde": "91501",
  "admn": "CA",
  "add1": "624 EAST SAN JOSE AVE UNIT 106",
  "add2": "BURBANK",
  "lcty": "BURBANK",
  "ctry": "US"
 }
]
In the above example, duplicate data is found in Address Line 2 and Locality. If the DuplicateHandlingMask option is not enabled, then the address is impacted in the following ways:
  • The address cleanser does not check for duplicates in the input address.
  • This affects the accuracy of the address cleansing process.
  • The address is not verified as the cleanser cannot eliminate valid duplicates found within the address.
  • Address Verification Code (AVC) is V42-I55-P5-089.
Note: Duplicates in input addresses can even lead to the rejection of the address by the address cleanser.

Example of Address with Duplicate Data - with the DuplicateHandlingMask Option Enabled

[
 {
  "pcde": "91501",
  "admn": "CA",
  "add1": "624 EAST SAN JOSE AVE UNIT 106",
  "add2": "BURBANK",
  "ctry": "US"
 }
]
In the above example, once the DuplicateHandlingMask option is enabled, the accuracy of the address cleansing process is enhanced. After the DuplicateHandlingMask option is enabled, the address is impacted in the following ways:
  • Duplicate addresses are compared across all levels of inputs and eliminated if found as valid duplicate addresses in both pre-process and post process stages.
  • The accuracy of the address cleansing process is enhanced.
  • Address Verification Code (AVC) is updated to V55-I55-P7-100.

Adding the DuplicateHandlingMask Option in the L3 Configuration

Like the other server options, you can enable the DuplicateHandlingMask option in one of the following ways:
  • In the Physical Configuration - Contact the Support Team to enable a server option at the physical configuration level.
  • In the L3 Configuration - See Configuring the Server Options to configure a server option in your L3.
Note: Following are some useful information about the DuplicateHandlingMask option:
  • The DuplicateHandlingMask option can be enabled in the L3 configuration. However, to enable this option in the L3 configuration, it must be entered in the option across each of the entity types that have an address.
  • Your L3 configuration overrides the physical configuration.
  • After enabling the DuplicateHandlingMask option, re-cleanse the addresses to ensure that the enabled option is applied and affects the existing entity types. Prior to enabling this option, few of the addresses that were earlier partially verified might get verified.
  • The process of cleansing can have an effect on the matching rate, as the standardized addresses that were not matching, can match and eventually merge.
Sample L3 Configuration with the DuplicateHandlingMask Option Enabled
{
"infos": [
 {
  "uri": "configuration/entityTypes/CleanserCrossWalkTest/cleanse/infos/default",
  "useInCleansing": true,
  "sequence": [
   {
    "chain": [
     {
      "cleanseFunction": "Loqate",
      "resultingValuesSourceTypeUri": "configuration/sources/ReltioCleanser",
      "proceedOnSuccess": true,
      "proceedOnFailure": false,
      "mapping": {
       ...
      },
      "params": {
       "returnUnverifiedStatus": "true",
       "loqateDefaultCountry": "US",
       "ovOnly": "true",
       "opts": {
       "PreferPrimaryValidAlias": "Yes",
        "OutputCasing": "Upper",
        "DefaultCountry": "IN",
        "DuplicateHandlingMask": "15"
       }
      }
     }
    ]
   }
  ]
 }

Activating the DuplicateHandlingMask Option

You can specify the various levels at which you want the duplicates to be verified. These levels are set in between 0 and 15 with 0 being the default value. You can activate the DuplicateHandlingMask option at the following levels:
  • Single Field Level: If the DuplicateHandlingMask option is set to 1, then the address cleanser pre-processes the address input and removes the duplicates that occur within a single field.
  • Cross Field Level: If the DuplicateHandlingMask option is set to 2, then the address cleanser pre-processes the address input and removes the duplicates across all the fields.
  • Tag Field Level: If the DuplicateHandlingMask option is set to 4, the address cleanser pre-processes the address input and removes the duplicates in fields, which have not been internally tagged. An input field is considered a tagged field if the field name is one of the standard address component field names such as Locality, AdministrativeArea, Thoroughfare, and so on.
  • Field Status Level: If the DuplicateHandlingMask option is set to 8, the address cleanser post-processes the output from verification and removes the duplicates from the non-verified fields.
Note: The values for this option are non-exclusive and hence multiple combinations are supported. To achieve optimum results we recommend that you enable all the combinations. For best results, the value must be set to 15 to check for valid duplicates across all levels.