It always intrigues me to see how people analyse results from a data project. Broadly speaking, people fall into two main groups:
1. Those who look at what has not been done
2. Those who look at what has been done
Both these techniques are good and when both are applied great results happen.
The problem comes when you only look at data one way. Let me explain.
If you have someone who looks at data by looking only for problems (or exceptions) then you can lose sight of what’s been achieved.
In fact, I came across a client that only did this. Processing several million records and focusing on the 2.5% that caused problems can be disheartening, because the original data quality was about 30% and to increase to 97.5% was considered to be a great result.
However, the client saw 2.5% of a big number, say 5 million, which is 125,000 records that could not be fixed.
The number looks huge, but there are very valid reasons why this set of records could not be fixed. In this case the data was very sparse and almost impossible to make sense of.
Let’s not forget the improvement from 30% to 97.5%, which is 325%.
Having a client that focuses on exceptions can be very useful, because we start to look at the problem areas and it is very easy to eye-ball data that’s not right. The challenging approach makes for better innovative approaches to solving the problem.
But, there comes a point where you only have diminishing returns for your efforts.
Talking to suppliers only about problems isn’t the best way to manage them.
Appreciating the success and focusing on the exception is good for relationships with the data team as well as motivating them to achieve a better result.
If you have a client that isn’t interested in the exceptions, but sees the positive results, then that’s great for the data team (no difficult discussions), but further progress isn’t made because the focus on exceptions is missing.
From a supplier point of view, you like to have both types of analysis. The client knows the good work that is done and knows that is probably the best it can be.
Now with every client of mine I make a point of ensuring that they do understand how to analyse the results, what can be achieved with certain types of data and how to measure it.