In my first installment of this series I wrote about writing a great problem statement, which is arguably the foundation of all Lean Six Sigma (LSS) projects. With a great problem statement developed, the next step is determining how best to measure the problem; what I characterize as a “winning” number. In this post I’ll discuss how to pick a winning number for your LSS project, which begins with a question-how can I determine the right metric to use for my project?
Before getting into how to pick a winning number I want to share my thoughts on some of the problems the process of picking the right metrics creates when not done properly. Most often the problem I encounter with LSS professionals is that we try to measure too many things in a project.
We often believe the more charts and graphs we can use the better the project; in some ways the more complicated we can make a project the more we tend to feel we are worthy of calling ourselves Black Belts, Master Black Belts, etc. There’s nothing wrong with getting excited about data, in fact that is one way to select great belt candidates, but most of the world doesn’t run to data and statistics, they run from them!
Another challenge faced with metrics is that it is often times hard to get the data. Rarely do I work on projects where the data is readily available. Usually the data has to be collected, or at a minimum, mined from a data source, which in many cases means getting people to do something new, and any time we ask for a change in behavior (i.e. collecting or mining data) we can expect a challenge if we don’t consider the “what’s in it for me” (WIIFM) factor.
Additional challenges I frequently encounter include converting process data to dollars, getting adequate sample sizes, and having a tight link between the problem statement and the primary metric.
What is a “winning” number?
Before getting into the process of how to pick a winning number it’s important to understand what characteristics describe a winning number (aka primary metric). A winning number should be:
- Linked to the problem statement
- The number that best represents the magnitude of the problem
- “Normalized” to avoid misinterpretation of ups and downs that mean nothing
- The number, that if it goes up / down significantly, will confirm the solutions have had a positive impact on the problem
- Easy to convert from process measures into financial performance
First and foremost, a winning number has to be tightly linked to the problem statement. For example, if you are focused on an efficiency problem your winning number should not be a measure of defects. This seems like a no-brainer, but it never ceases to amaze me how often the numbers we come up with don’t link to the heart of the problem we are trying to solve. Sure, quality will no doubt impact efficiency, but it’s not how we best measure it.
The number also has to provide a sense of how big the problem is. In other words, how best can the problem be converted into a number that quantitatively illustrates to size of the issue?
Normalization can be confusing so an illustration works best to describe this idea. Look at the charts illustrated below. Do you see any differences between them?
Based solely on chart 1 we may assume the process has a lot of variation, but put into perspective by looking at the data in chart 2 we see there is virtually no variation in the process. Essentially, normalizing the data keeps us from reacting to ups and downs that have no meaning.
Another way to pick a winning number is to ensure the number we have selected gives us the best answer to the question leading to determining if the solutions have indeed led to a quantifiable improvement. In other words, if this number goes up / down is that a good indication the problem is now less of an issue?
Finally, none of the improvements mean much to leadership if they don’t lead to financial gains. Converting process data into dollars can be tricky, but it’s not impossible if we start with the end in mind.
A winning number should be easily convertible from process performance to financial performance. For example, if the process is measured in time what is the time worth? Another common example is defects and determining what a defect is worth. Sad but true, if we can’t make the connection to dollars you are unlikely to see long term engagement in the LSS process by senior leadership. Business performance (for-profit) is ultimately based on financial performance not process metrics.
How to Pick a Winning Number
Picking a winning number can be done in a number of ways from simple brainstorming to a more structured approach using evaluation matrices and scoring mechanisms. I propose a hybrid approach that goes beyond brainstorming, but doesn’t become overly complicated with unnecessary rigor.
1. Start with a simple individual brainstorming session using post-it notes to capture ideas. I like to initiate the session by asking a simple question such as, “how could we measure our problem?”
2. Group the ideas into categories based on what they would measure. Typically, all LSS projects fall into one of four categories such as a) productivity, efficiency, and speed, b) quality, customer service, and reliability, c) cost, revenue, and profitability, and d) safety, environmental, and health.
3. Determine which of the aforementioned categories the problem statement falls into and zoom in on those ideas. Affinitize (group together) similar ideas within the focus category.
You could also start with evaluating the problem statement and “forcing” the group to only brainstorm in that space (i.e. productivity, efficiency, and speed), but I find that opening the process up to all ideas leads to a better outcome and more ideas, and by doing so you can also create a teaching moment on other types of metrics for problems in the other categories. Remember more is better when it comes to brainstorming!
4. Use the following questions to find a potential winning number:
- Does this data already exists*?
- Assuming the data exists, is it accurate and reliable, and do people trust the numbers?
- Are enough samples available?
- Can the data be easily converted into dollars?
*If the data does not exists I deeply discount it as a winning number because of the challenges of getting the data. I find that it is far better to use “good enough” existing data than it is to collect “perfect” non-existent data.
From these answers the best number(s) is the generally one that already exists, and is both accurate, reliable, and is trusted by others. I have found that even though data is accurate and reliable if people in the organization find the numbers to be “funny” numbers no amount of accuracy and / or reliability will get them to buy into the data.
Sample size can also be a challenge, especially for transactional processes. You will need to determine how to define an adequate sample size, but in general I’m satisfied if I can get 20-30 data points evenly distributed between the Measure and Improve phase.
Finally, if the number can be converted into dollars you may have found a winning number to measure your project!
Picking a winning number can be challenging, but it doesn’t have to be overly complicated if you follow the steps outlined in this post. Some additional thoughts on what makes a winning number include focusing on the positive aspect of the metric such as whether to measure defects or accepted products, transactions, etc.
I highly recommend always focusing on the positive side of the number simply because it makes for easier positive reinforcement of the behavior that leads to a higher number. How rewarding does it feel if your boss comes around after a solution has been implemented and says, “great job keeping your screw ups to less than 5%”. I’m sure you’d much rather hear, “great job keeping the acceptance rate at 95%!”.
Another consideration is when having to collect data get those who will be involved in doing the long term collection doing the data collection in the Measure phase so that when you get to Control there is no hand-off to be made. Those who were collecting the data in the beginning simply continue doing so for as long as needed. This will minimize the probability of failure after your team steps away from the project. For more about the behavioral side of process improvement read about the ABC’s in a previous post here.