Normalizing Country/State: The Contact Washing Machine

This post is a continuation of the post on normalizing the titles of contacts in your database. The goal of these articles is to provide some direction on how you can create an automated process to clean up your data which can assist you with segmentation and lead qualification - essentially helping you get more out of your database. This post will specifically focus on cleaning up country and state/province fields but you'll be provided with a method that you can use to clean up other fields. If you have a database that only has United States in one format, I salute you. For the rest of us, you need an easy method to help cleanse your data and help you build more targeted lists. As a quick summary, you're going to create a program that will look for incorrect values in the contacts that flow through a program that will hopefully correct those values. 

To learn more about the concept of the Contact Washing Machine, see Why the Contact Washing Machine must be In-House. This automated process is not just something that we preach about here at Eloqua but we have built out our own Contact Washing Machine (and probably more like a dry cleaning factory) and live by this concept of clean data. I encourage you to check out Data Management and Marketing Automation - Video to see how we do this here at Eloqua.

This post assumes that you are familiar with Eloqua. You may need to do some research in the Knowledgebase on some of the specifics of the features mentioned. In addition as it uses Program Builder, I would check out the video on Program Builder in Eloqua University or the courses that are available.

To begin this process you need to have analyzed the country or state/province field in your  database and determined that it's need to be cleaned up. One tip worth experimenting with is using a free tool called "Google Refine" to help examine your data and look for trends. It can help you determine that United States is spelled in 10 different ways. This is needed when creating your Contact Washing Machine.

Step by step instructions:

• Create a Data Cleansing File. The next step is to create a new CSV file that will contain all incorrect values in one column and the normalized values in the next column over (same row). In the example we used, the incorrect values are the different ways that Massachusetts was spelled and correct values are the ISO values of state. We're going to replace the incorrect version of Massachusetts with the ISO value. You don't need to do it this way - that's up to you.

You will also want to add a "Company Name" (or Account Name) column and use a standard value such as "Data Normalization - State". We do this because we're uploading this data to the Company/Account content type in Eloqua and this will help us identify these records if need be.

To make this process easy for you, we have provided some CSV files that you can use. In both examples, the CSV file will correct country and state to a two letter code. For example, United States will become US:

• Country Normalization

• State Normalization

TIP: There should be a record in the CSV for each field that is to be normalized (Zip, Country and Region, etc…).  In order to fully benefit from the Contact Washing Machine, you have to review what you already have in your database and update the sample CSV files provided. These files are not comprehensive and have not predicted all of the data entry issues that your company is experiencing. Before uploading these CSV files (e.g. for “State"), export your data to excel and compare the values you are trying to fix (e.g. State) in the CSV file (as suggested above you can use Google Refine to assist). Add values to the CSV file that are in your database but not in included in the suggested CSV file. For example, if you see "United States" spelled "United Stats" (missing the "e"), add that to the file. You can also label some junk values in your database “n/a” and perhaps remove or opt to fix this those records using another method. 

• Create new Company/Account field. Create an “Incorrect Value” field on the Company or Account table in Eloqua (the name depends on your Eloqua version) to store your incorrect values in the Company/Account content type area. You'll want to do this for every type of data that you correct (State, country etc...). For example, for State, we created the “ISO State Fix”. These fields will used within your Match Rule to append the normalized value (explained below).
  
  
• Create Company Groups. Create Company Groups in Eloqua for each CSV file that you've created (State, Country, Region etc...). This group is what you will reference within your Match Rule to normalize the data. For simplicity, call your Company Group "Data Normalization -State".
  
  

• Upload CSV Files. Upload your CSV file into a Company group in Eloqua you created and ensure that the mapping of the incorrect values go into the new Company Field that you created (in this example “ISO State Fix”).The correct value (ISO in the CSV example above) can be mapped to "State or Province" (or a similarly named field that already exists) and the Company Name should be mapped the Company Name field.

  You may need to repeat this process if you have additional criteria such as country etc...
  
  

   

  TIP: When you need to make changes to your business rules, we recommend that you make changes in your CSV file and re-upload your file to the Company Group. You'll need to delete the existing data in the Company Group first so you don't create duplicates.

   

   

  TIP: To easily see the normalized information in the Company table, create a Company/Account View that includes the fields that you are using to normalize your contact data. For example: Company Name, State/Province, ISO State Fix etc..

   

• Create a Match Rule. To update contacts with the correct state data we need to create a Match Rule that will identify and match the incorrect values that need to be fixed. We will be matching the incorrect value in the Contact record with the value we uploaded in the Company record. This match rule will be used as a “Step Action” within the Program Builder Contact Washing Machine we will create.

  Follow these steps:

  1. Create a Match Rule matches Contacts to Companies (default)
  2. Select "Exact Match" as the criteria
     1. Select the "State/Province" as the Contact field
     2. Select the "ISO State Fix" as the Company field
  3. Skip step 3 and call the Match Rule "Data Normalization - State Fix (Contacts)"

  This is what the Match Rule should look like:
  

  Repeat this step if you need Match Rules for other criteria such as country.For country, the match rule will be similar but the Contact field and Company fields will differ.

   

• Add Match Rule Handler Set. Now that the Match Rule is created, we want to define the rules on how to update the state field once we've identified a match between the Contact and the Company fields. This is done by adding a Match Rule Handler Set to the Match Rule we just created.  This rule will update a Contact field (incorrect values) with the data from a Company/Account field (normalized values).  These rules are to be built for each field that is to be normalized.

     Follow these steps:

1. Call the Handler Set "Update State Value In Contact"
2. Choose "Update a field with with field values from matched record" on Contacts
3. Choose "State or Province" as the field to update on the Contact record
4. Choose "State or Province" as the from field on the Company record.

     This is what the Handler Set should look like:

• Create Your Contact Washing Machine Program. Now you can create a Washing Machine Program and run your contacts through the program using Program Builder.   
  • Go to Program Builder and create a new program. Name the Program and choose "contacts" as the default member type.
  • Once you've created the program, add an Action step as the first step in the program but don't add any actions to it.
  • Now, add a new Action step to the first step.
    
    • Select the "Run a Match/Dedupe Rule" action
    • Run a Match/Dedupe Rule against "Companies"
    • Run a Match/Dedupe Rule against members in the Company Group you uploaded
    • Choose the Match Rule you created
    • Choose the Match Rule Handler Set you created

It should look like this:

• If you want to add an additional cleansing process such as a country cleansing Match Rule step, add it now.
• Once you are done, you can add a final step that will remove contacts from the program.

• Test. Test your Match Rules and make changes if needed to ensure that the majority of state/countries error types are accounted for so that they can are corrected. You may want to disable the final step in the program and review the data for the first few weeks. You may need to modify the records in your Company Group or upload new records based on your testing. Once you are confident that all is OK, let the contacts flow freely through the program.

TIP: To make adjustments, I recommend deleting all records in the Company Group, adjusting the original spreadsheet and re-uploading all records.

TIP (from Paulina Johnson): For the testing purposes, you may want to add an additional column to the State/Province CSV file called Email address. You can create test email addresses that will correspond to the number of the row of your future lookup. This will help when your testing certain scenarios to ensure that the data uploaded is correct. Here is an example:

Create a test Company Group in Eloqua “Data Normalization – State Test CG” - upload a CSV file that includes the test email addresses and associated ISO State Fix columns. This test data should match the CSV file. Run your test contacts through the program and then examine the results of the Group by exporting the the Company Group data and comparing it to the main CSV file to see if the data has been corrected as expected.

• Recommended: You can now add Program Feeders that will automatically pull in contacts and cleanse them. Here are some examples:

  • On your Eloqua forms, have the data flow directly to this program instead of to a CRM integration program.

  • Create a Program Feeder that adds new contacts to the program

• Recommended: You can also pass the data to your CRM integration program using the "Move to Another Program" program action:
  

  Your completed program may now look like this:
  

Remember to review your data and tweak the Company upload to account for new scenarios.

This is designed to show you about some of the data cleansing possibilities. Let us know about any other examples you have used to cleanse your data.