All posts by Neil Wuebker

Why does data quality matter?

Years ago I worked in the Financial Service industry. I worked for three different Credit Unions and came across data issues constantly. Sometimes it was as simple as a check that was unable to be deposited because the spelling of the name was incorrect, or it was made out to a business name rather than the owner. Other times it was as serious as an individual’s credit being ruined because they had the same name as a relative and the credit reporting bureaus recorded the relative’s loans and late payments on their credit history instead. On a side note cleaning up those incorrectly reported credit errors is very difficult, but has gotten easier, primarily because identity theft has become more frequent in our society due to that credit problems are more common. I think everyone can think of a reason why it’s important for a consumer to have correct data, but it’s not always clear why businesses need to make every effort to keep their data clean and free of errors.

Data errors can affect any aspect of business. Everything from product stock, to financial numbers, to marketing effectiveness is tracked with data. If any of this data is figured incorrectly serious issues arise. Companies need to evaluate their systems and find the areas that are most vulnerable to data errors. One major area is Customer information. Unlike other areas, customer data is constantly changing, addresses, phone numbers and credit card information are often needing verification and updating. These are some of the most difficult areas to keep correct, because unlike data that a company provides, these types of data are completely reliant on outside sources. Businesses need to have the right tools in the hands of the right individuals to decrease the frequency of inaccurate data across every area of business.

All data is not created equal.

Last week Toyota announced that they would end production on the Scion brand of cars. Thirteen years ago Toyota launched Scion because their data showed that small cars were a good bet. Times were different in 2003. The US was still adapting to a post 9/11 world. Gas prices were on the rise and the economy was recovering. Jobs were good for college graduates and Toyota wanted to offer a product that 20 somethings could afford, and would be a good fit for their lifestyle. Their data suggested that Scion was a good investment. ScionFor years Scion made money and did decently. In 2007 the economy took another tumble. The middle class got hit really hard. This caused Scion to lose more sales because their target demographic took the biggest hit financially because of the economic down turn. Eventually Toyota, by teaming with Subaru to build the FR-S/BRZ, strived to change public opinion of Scion as a small inexpensive car brand for the 20 something crowd. Even after these efforts Scion failed to see much improvement in sales. While no company can see the future, if Toyota had collected more data, and adapted Scion’s business plan to include a small SUV when gas prices started to drop, or planned on adapting their small cars to accommodate two adults and a child seat comfortably, then they may have been able to increase their market share in the US. In the end Toyota chose to cut their losses and move on.

What can be learned from these business decisions? While lots of data can be collected through several sources, the strength of that data needs to be weighted. In addition the quality and source of the data are important aspects to be considered too. In the example of Scion, maybe Toyota could have given the lowering gas prices more importance and built larger vehicles to accommodate more people comfortably and traded off for a lower MPG. Or they could have focused on the losses the middle class were experiencing and built a luxury sedan that focused on a higher income demographic. In the end, they may have feared competing too closely with their other two automobile brands and just kept doing the same thing hoping for the best.

Self Service Analytics

Recently articles about data analytics have  pointed to the need for products that are designed for the Self-Service user. Historically data analytics have been done by data professionals or IT personnel with temperamental, complicated tools. In a recent article by Daniel Gutierrez, discussing the future of data analytics one point that he brings up is the changing trends and benefits of self-service platforms that enable the everyday user to function as data analysts.empower-with-data

Many companies have built visually appealing and informative features into their software trying to empower less-experienced users to easily analyze and identify trends in their data with the goal of helping them derive new strategies for business. This can be helpful at times, but sometimes just makes things even more difficult to understand. While it is good to enable business users to analyze data, if the data is corrupted or out of date users are still left in the dark. By building queries on a foundation of accurate data and addressing data quality initially users are able to reach more accurate conclusions. This is why at Aim-Smart, we believe that giving business/self-service users the power to match, deduplicate and verify the quality of data is a necessary first step for their data analysis.

By creating a tool with the goal of achieving data quality in Excel, Business/Self-Service users are able to clean, match and deduplicate data in a familiar environment. This decreases the learning curve of a new product for less experienced users. In addition it allows users to manipulate the data by using the existing Excel features they are familiar with. By allowing users to build their queries on a clean data foundation they can reach conclusions with greater accuracy. Another benefit is that this removes the need to move data back and forth with IT personnel for cleansing and verification, this reduces possible data corruption points.

Is your company ready for the future of data management? Have you taken the steps necessary to empower your employees to make business more successful?

Excel 2016 Implementation

excel-2016-logo-595Our team at Aim-Smart is excited to announce that after diligently working over the past months, Aim-Smart is fully compatible with Microsoft Excel 2016. With the updated version users are able to take the well-known abilities of Excel to the next level. By adding Data Quality features from Aim-Smart. Excel 2016 is enhanced with the tools any business user needs to deal with large data sets. Thanks to features that include the ability to fuzzy match entries, and process high level deduplication, Aim-Smart really boosts the abilities of Excel 2016. Optional features also enable users to do address verification with in Excel 2016. For more information about address verification using Aim-Smart in Excel read our blog post here.

Does your company have “Big Data”?

During the past few years business professionals have heard the term “Big Data” emerge. At times is has been used both positively and negatively. Some companies tout their “solutions, using Big Data” others use the term as a scare tactic “How will your company deal with Big Data”. Before worrying about what to do with Big Data, let’s first discuss if your company has Big Data.

Big Data QuestionsBig Data is defined by three aspects, Volume, Varity and Velocity. Volume is the amount of data that is being analyzed, Varity is the number of fields of within the collected data. Velocity relates to the speed at which data is being collected, real time data collection is becoming more and more common. While all these things would be very useful to most businesses, the reality is, most businesses don’t have the financial ability to collect and process this much data. In addition many businesses don’t require Big Data to thrive and grow in their marketplace.

Of course there are large corporations who process and need Big Data, but your company may be just as effective with much smaller amount of information. In these situations products offered by large data analysis companies can be overwhelming and too broad for your needs. In addition they can be more expensive than your company’s budget can afford. It has become evident in the current business market place that more manageable products are necessary to fill these needs. Smaller software products that utilize data quality in Excel are a great place to start. Products that produce results with amounts of information much smaller than “Big Data”, yet have the ability to process increasingly amounts of data as a company grows are very useful.

Aim-Smart is an excellent option for any business regardless of size. Aim-Smart is easy for business professionals and management with little to no data analyses experience to use. In addition, when used properly, Aim-Smart eliminates incorrect data and prevent incorrect data from lowering the quality of analyzation results. By placing these abilities in practically any employee’s hands team members are more informed and will make better decisions about what to do when the company has an issue that data can solve.

Finding cost-effective solutions to data gathering and maintenance for small businesses

The U.S. loves small businesses. I would argue that every individual American has at least one small business that they love. These businesses vary in types and varieties. Some individuals will tell you about their favorite local restaurant, they will praise their small farmer’s market or a local coffee and doughnut shop. These small businesses are what modern America was built on. Today it is becoming harder and harder for these businesses grow and thrive.

Here are some questions that concern small business owners.Small Business

1. How can we build and manage our data when most of the products are priced way out of reach?
2. What first steps should we take to better our ability to deal with data?
3. Where will my dollars make the largest difference?

While large firms that specialize in data have huge price tags, there are some reasonably priced alternatives for small businesses. Database software is becoming more reasonable in price, in part because of downward pressure coming from MySQL and other open source systems. A business owner can maintain their own database with a small amount of training or find technical contractors who, for a reasonable price, will setup and query databases for any business. In the past there has been one thing missing – good data quality software for the small business. That’s why we helped create Aim-Smart for cleaning and removing duplicates from their existing and incoming data.

In addition to having reliable places to store and manage the data, businesses need effective ways to collect data. Many businesses offer incentives for customers that are willing to share their data. These include email coupons, club memberships that come with discounts, or just offering birthday incentives. Building up your database of customer information is very important. Keeping customers up-to-date with emails and mailers keeps you fresh in their minds and keeps them coming back.

It is a well-known fact in business that maintaining customers is considerably less expensive than finding new ones. Depending on the industry it costs 4-10 times more money to find new customers than to keep existing ones. This means every dollar you spend to maintain contact and better your relationship with your current clients is far more valuable. Having customer data up-to-date is vital to this. Small businesses can find address verification software that is reasonably priced and easy to use. Once they have found an inexpensive database process, they can easily maintain it themselves with minor effort and training. In addition, Aim-Smart offers a National Change of Address service whereby customers can update their customers’ addresses automatically, should they move. This leverages the USPS change of address system (those little cards you fill out when you move so that your mail gets forwarded).

Thanks to new software options and prices small businesses can afford, smaller businesses have greater ability to compete against their larger rivals.

Helping small and medium sized businesses compete with large business analytics.

Small BusinessWith the New Year upon us it is a great time to assess where business can become more efficient. The ability business has to process analytics is expanding faster and faster. Thanks to modern computers and advanced software, interested parties can collect and evaluate more customer data faster than ever before. Businesses have to be cautious though. There are holes that cause issues with the results data supplies, if that data isn’t as correct as possible. Since the economy crash a few years ago, an increasing number of customers have rapidly changing information. Many people who were once home owners, now rent, and move with increased frequency. Many are changing phones plans and phone numbers. Others are getting married or pass away. There are far fewer constants today than in the past. As companies collect more data, it becomes more difficult to verify if their data acquired in the past is accurate. This creates multiple records for the same customer with varying levels of accuracy. These records not only fill up company databases, but keeping incorrect data wastes company resources. Large companies employ professional companies or firms, or they use department resources to monitor their data to and remove inaccurate information.

We know that large companies are greatly affected by incorrect data. Poor data has caused several large companies to make decisions that eventually removed them from the current market place. Companies like Kodak, RCA, and Motorola who were once Fortune 500 companies are no longer in business. If large companies can be ruined by poor decisions, what does that say about the need for small and medium sized companies to base business decision on accurate data?

In today’s business environment small and medium businesses struggle from limited access to effective tools for dealing with data. Big businesses can afford expensive software products, making it difficult for small and medium sized business to compete. But what about the little guy? This is where Aim-Smart comes in. Aim-Smart is a powerful Excel data quality solution for small and medium sized companies. Not only is it fairly priced, it allows small/medium businesses to leverage best-in-class data matching, address validation, and data analysis within Excel.

What will your company do this year to increase data quality?

Reduced Pricing for the month of December!

Aim-Smart is now offering a new purchasing option and reduced pricing.

Aim-Smart is introducing the “Total Company License”. This is a great option for companies who have multiple users in multiple departments that can benefit from the high level data matching Aim-Smart offers.

Aim-Smart’s current price is $10K USD for a single license which covers one computer installation but allows for multiple users on that computer. Aim-Smart also offers a 5 license bundle for $35K USD and, like the single license option, allows for multiple users on each computer. The 5 license package is popular with companies who have multiple departments who use Aim-Smart but aren’t physically close to one another. By installing Aim-Smart on a shared computer users can receive their results easily and save the files for access later.

During the month of December we are offering reduced prices on both the single and 5 license packages. Please call for details. 303-772-6100

The “Total Company License” Now companies have the option of installing Aim-Smart* on all of their machines for a deeply discounted price compared to the single and 5 license packages. Call and get a quote today! 303-772-6100

*some restrictions apply, contact Neil at ext.145 for further details

Why is data cleansing important to companies?

The U.S. economy wastes an estimated $3 Trillion per year due to incorrect, inconsistent, fraudulent and redundant data. Businesses incur a very large portion of this cost. The Data Warehousing Institute estimate that the total cost to U.S. business more than $600 Billion a year from bad data alone. That is a very large cost that can easily be reduced by investing a much smaller amount of money earlier business data processing. No business is immune to financial loss from lost sales or extra expenses caused by incorrect data. However businesses are able to limit these losses through proper data cleansing.

The rule of 1:10:100 by W. Edwards Deming says that it takes $1 to verify a record when it is first entered, $10 to clean or deduplicate bad data after entry, and $100 per record if nothing is done to rectify a data issue. Part of this cost is incurred due to mail cost. The average company wastes $180,000 per year on direct mail that does not reach the intended recipient due to inaccurate data. Poorly marketed advertisements to incorrect demographics are another expense of having poor or out of date information.

The most common issues found due to data quality are duplicate and old records. Within one month of receiving customer data 2% becomes incorrect due to moving, death, marriage or divorce. If you are receiving data from an outside source, i.e. other departments, other businesses, these sources may not know how old the records are, if imported data is several months old 10% or more may be inaccurate. By having tools in house, companies can limit the level of incorrect data they deal with. An issues with in house data cleaning is that IT departments are often heavily relied on for that process. IT professionals have expert knowledge in technical issues, but often lack in-depth data processing best practices. Usually companies employ data analysis experts, but again, they often lack in-depth IT knowledge. The strength of utilizing business data is dependent on these professionals ability to function at their best. Often there is time lost or wasted due to lack of clear communication between these two departments. In addition lack of understanding causes incorrect information to be returned and can result in having to redo various aspects of the process.

What does Aim-Smart do to eliminate these issues? Aim-Smart is built with the business user, analysis expert and IT personal in mind. With Aim-Smart business users and analysis experts are able to manipulate data with the speed and accuracy of IT professionals and use their knowledge of advanced data processing techniques to remove or update data as needed. This is done all with in Excel, a comfortable environment for any business professional. This allows the IT professional to focus on the upkeep of the data platforms and focus on maintaining the platforms for data storage. I also removes many opportunities for misunderstanding and communication breakdown. It is estimated that up to 50% of IT expenses are spent on data cleaning. Aim-Smart removes most of the data cleaning process from the IT department. By putting the power of data cleaning in the hands of the business user, IT demands drop and allows the IT professionals to focus on strengthening and updating internal systems within the company. Aim-Smart allows business users to remove data, or update records they find to be out of date. It also allows them to manipulate the data as they want through parsing, standardization and other features. By doing this, company marketing budgets can more effectively be spent on contacts or data results that are accurate. This helps to maximize the use of company money spent on advertising as well as other expenses.

“$3 Trillion Problem: Three Best Practices for Today’s Dirty Data Pandemic” by Hollis Tibbetts

How Deduplicate is Superior to Excel’s Remove Duplicates

Excel includes a standard duplicate removal tool called “Remove Duplicates”. I have used this tool before matching records using Smart Match (to avoid duplicate matching). While Remove Duplicates is useful, there are two fundamental issues to be aware of.

First, Remove Duplicates only removes exact matches within the Excel Sheet. On several occasions I’ve had two records that are the same, but because they come from different sources they aren’t exactly the same. One record may contain a contact with the name Anthony, but the same contact may be listed in another record as Tony. Remove Duplicates doesn’t give me any options for finding duplicates that aren’t exact matches.

The second issue I have with Remove Duplicates is the information it provides after removing the duplicates. The only information a user receives is the total number of removed records. Remove Duplicates won’t specify whether one record was duplicated several times or if several records were duplicated only once, or if there was some combination in between.

Deduplicate addresses these issues. When running Deduplicate I can access all the power of Aim-Smart’s matching logic, which allows for superior deduplication identification and reporting.

Here is an example of where Deduplicate results differ from Remove Duplicates. When using Remove Duplicates users have no ability to choose if duplicates are removed or not; in a way this isn’t a problem because every field matches exactly, so we know that we want the record removed. However, it leaves several possible duplicates (that vary by a small amount) untouched in the document. When using Deduplicate with the intelligent deduplicate logic we can see multiple sets of exact duplicates and we can see varying quality levels of other duplicates as well. This is useful when data may look like this.DeduplicateOriginal

Using Remove Duplicates on all columns, with the data provided, would only remove row 4 and leave Rows 2 and 5, which are of course real duplicates. Remove Duplicates allows us to choose limited columns to compare. If a user chose a column subset (Last, Address, State) then rows 2-5 would be removed, but row 3 is not actually a duplicate. Without Aim-Smart’s Deduplicate tool it would take a lot of time and effort to identify (and potentially remove) the approximate match duplicates.

Deduplicate is different. Users can enter several different deduplication options at the same time. For the case above one option would be to create deduplication parameters like these.Blog Screen Shot

With the options shown above Deduplicate will return the following results.DeduplicateResults

With these results a user is able to assess which matches are true matches and which ones aren’t true matches. The user then knows based on the [ID] which rows to remove in the original document. In addition users can easily see in the results how many duplicates each record group contains.

Thanks to these abilities and features Deduplicate is a more powerful tool than Excel’s Remove Duplicates tool.