Can I Trust My Data?

When TMAC staff visit a company for the first time we ask a series of questions as part of an informal assessment of their business. One of the most basic questions asked: How do you use the Measurement System at your company?  Do you use it to separate good from bad?  Do you use it because it’s just a step in the business process to provide a product or render a service to your customer? Is the act of measurement being done a certain way because “That’s the way we’ve always done it?”.  

The real question the TMACer is asking: Can the data be trusted? If the answer is ‘No’ then anything communicated involving data during the site visit must be taken with a grain of salt. We have seen too many instances where bad data led to bad conclusions. And in some cases, bad results for the business.

All LSS practitioners know the DMAIC process uses a structured, data-based approach for problem solving. It prescribes use of a Measurement System to learn more about a process in order to establish a baseline (Measure Phase), determine root cause (Analyze Phase), confirm results from a pilot (Improve Phase), and compare the changes implemented in the process to the baseline (Control Phase).  In other words, data are used throughout the life of a LSS project. Of course, without trustworthy data none of this can be accomplished.

During a site visit there is a related question we sometimes ask: “What do you know about your Measurement System process?”  Basically, we are trying to determine how much company management knows about the methods put in place to measure critical business processes. The response helps us ascertain if management trusts the data at their company. 

The first thing I think management should know about their measurement processes is tied to some basics of any business enterprise: A firm’s procedures and their training program. Here is a list of related questions we may ask:

• Do work instructions or procedures exist on how to perform measurements? Are they adequate, given customer requirements?
• Has the organization spent the time to teach their staff the proper methods to make accurate, precise, and consistent measurements? 
• If so, are staff following these procedures?
• Do they keep training records of staff performing these measurements?
• In some situations, we may ask if there is a formal program for certifying employees trusted to perform these measurements?
• Do they have a program for ongoing calibration of all measuring devices?
• Are any devices damaged or in need of calibration?

The second thing to be checked is tied directly to the questions noted above: Does the measurement system provide accurate and precise data? Here is where a formal Gage R&R Study (for continuous data) or Kappa Study (attribute data) is appropriate. These types of studies help determine if there are problems with the measurement system, and what needs to be done to address those problems.

To illustrate my point, I will share a story about a past TMAC customer. Several years ago, I was working with a Black Belt on a project to improve quality. We determined that the measurement system may be culprit in some unsatisfactory conformance data.  After digging deeper, we decided a Gage R&R study was needed. As an outcome from this study it was determined that the three inspectors who participated in the study reported different measurements when measuring the exact same part.  That’s troubling, but in fact, it’s gets worse.  When providing these three operators the same part for a second round of measurements they each got a different value.  THE SAME PART.

What to do?

In this case the Gage R&R study results indicated the measuring device was just fine. The issue was inconsistency in the methods used by the operators. I worked with the laboratory manager to update their measurement procedures to make sure they provided adequate detail. Next, we held a training session with all of the staff working in the laboratory. Finally, we ran a second Gage R&R study which indicated the measurement system was now acceptable.

In sum, the fundamental question all managers should ask: Are we willing to make business decisions based on the determinations of our measurement systems?  For any organization, the answer should be yes.  However, if the two issues noted above exist then any decisions made could very well be wrong. For instance, if the measurement system indicates a part is good and we pass it on to the customer when in reality it is bad then in statistical terms we’ve just made a Type 2 Error ( Consumer Risk ) since the consumer will be at risk.  But that’s not the only risk at stake.  If we incorrectly reject a good item, we’ve just made a Type 1 Error ( Producer Risk ) since the producer has just incurred the time and cost associated with scrapping a good product.

Bottom line: Always check the measurement system. The LSS Methodology addresses these potential pitfalls through the teaching of the MSA or Measurement System Analysis.  Through tools such as Gage R&R and Kappa Studies, an organization can determine whether or not their Measurement System can be trusted to provide data which can be used to make business decisions.

Finally, to quote Dr. W. Edwards Deming: “Without data you’re just another person with an opinion”.