Can I Trust My Measurement System? Part 1

TMAC customers contact us periodically to help assess their measurement system. Past examples include a tier-1 supplier for the automotive industry, a tier-1 supplier for a telecommunication company, and a paper manufacturer. A current customer in the healthcare industry is launching a new product and would like to know if their measurement system is reliable.

What do these companies have in common? They are trying to clearly and reliably answer the question: Can I Trust My Data?  In this multi-part blog, we will begin by sharing TMAC’s approach to performing Measurement System Analysis. Our focus will be on conducting a Gage R&R Study for continuous data. However, the approach is basically the same for performing other types of MSAs (e.g., Kappa Studies for attribute data). In part 2, we will discuss how to analyze results of a Gage R&R study. In Part 3 we will share a case study of a Gage R&R at a TMAC customer.

Although measurement systems vary widely, we have found a consistent approach yields consistent results. Once we begin working with a customer, we start by going to the Gemba to observe the process, ask questions, and learn about the measurement system. We also ask a series of basic questions including: Does a standard exist? Have operators been trained? Has the measuring device been calibrated? What problems have you been experiencing with the measurement system?

Second, we use a kaizen model to conduct the actual Gage R&R study. As with any kaizen, planning is of critical importance. Typically, the study takes two days to complete. Engaging with front-line staff is one of the keys to success. Below we will discuss the steps to conduct a Gage R&R in detail.

Lean Six Sigma practitioners know validation of the Measurement System is a requirement of the Measure Phase. Hence, all LSS belts must perform an MSA on each project.  If the MSA passes the belt can proceed. But if the MSA fails, the belt must fix the Measurement System. 

What some practitioners forget is that a second MSA must be performed in the Control Phase. The focus of the MSA in the Measure Phase is on the output variable (Y). But for the Control Phase the MSA should be conducted on the critical input or process variables (Xs). Bottom line: If you use the DMAIC process, sooner or later you must conduct an MSA.

One thing we have learned over time: Each measurement system is unique. Hence, if the MSA fails for one measurement system the solution to fix it may not apply to other systems.  This is why an effective MSA is so important. Done correctly it will help you understand the root cause of problems uncovered during the study.

An important consideration in planning a such a study is whether the measurement is destructive or not. For example, measuring the length of a part is a nondestructive test. But measuring the strength of a weld requires pulling it apart – a destructive test. The general approach outlined below is the same for both types of tests. However, there are some special considerations for conducting a Gage R&R for a destructive test. For the purpose of this blog, we will limit our discussion to nondestructive tests.

Here are the recommended steps to Plan a Gage R&R study:

1. Select a particular part number for the study.
2. Determine the part characteristic to be measured (e.g., diameter of a hole).
3. Determine operators for the study. Should be 2-3 individuals who normally conduct the measurements.
4. Check to confirm the measuring instrument is calibrated. If it is not calibrated, we recommend having it calibrated before moving forward.
5. Choose 5 to 10 parts for the study.  In choosing these parts it is critical that they have a range of variation representative of the actual process.  Make sure to select a mix of clearly good and bad parts, plus parts that are partially good and partially bad. See the graphic below.
6. If historical data are available, calculate the process standard deviation.
7. Determine the specification limits. 
8. Determine how the study will be conducted in terms of operators actually measuring parts. There are two goals. First, the parts should be measured in a random sequence. Second, the operators should be ‘blind’ to the part numbers they are measuring. These precautions are taken to limit the risk of bias.
   1. Randomizing Parts: Most statistical software packages can generate a data entry sheet with a random sequence of both the operators and part numbers (see graphic below).
   1. Blinding Parts: To keep parts from being identified by study participants the parts should not be marked or numbered. The person facilitating the study must decide how to keep track of each part such that operators are unaware of the specific part. One way is to keep the parts hidden until they are measured.

All of these preparatory activities take place prior to the kaizen event.

Some additional planning considerations:  Should we use 2 or 3 operators? Should we do 2 or 3 replicates? The Automotive Industry Action Group (AIAG) established a standard for this type of study. MSA-4 is the AIAG standard which defines a Short Format Gage R&R as 2 operators, 2 replicates and 5 parts. A Long Format Study has 3 operators, 3 replicates and 10 parts. You aren’t limited to these two options and can choose something in between, such as 7 parts.   

Other practical considerations which impact planning a Gage R&R study:

• Is the measurement conducted by an outside laboratory? If so, what is the cost and turnaround time to get results?
• How long does it take an operator to measure one part? 
• Is there an activity required to prepare each sample before measurement? If so, how much time is required? Is additional time required after measurement occurs?
• Must production be interrupted to collect a sample? Or to conduct a measurement?

Here are the recommended steps to Conduct a Gage R&R study:

1. Before the study begins meet with all the key stakeholders: Operators, Lab Manager, Quality Manager, etc. Explain what is a Gage R&R study, the importance of validating a measurement system, and why this particular measurement system is being studied. This may entail sharing past results. Present training on basic statistical concepts. Then present measurement system concepts.
2. Explain how the study is to be conducted. Make it clear operators are to measure parts the way they normally do.  Answer any questions.
3. Get the parts, measuring device, sample bags, and any other materials ready for the study. Steps 1 to 3 will typically take up Day 1 of the kaizen event.
4. Have one operator measure all parts in random order. Then bring another operator and measure the same parts again in a different random order. Repeat until all operators have measured all parts one time. This is one replicate.
5. Next have the operators repeat the entire process a second time. This is the second replicate. A third replicate may be performed with all parts measured in a random sequence again.
6. As each operator measures a different part enter the data in the Gage R&R template. It is critical to maintain the blind, random format throughout the study. It is also critical to enter the data correctly.
7. Analyze the results.
8. Report results to the stakeholders and make recommendations, if any, to improve the measurement system. Steps 4 to 8 will take up Day 2 of the kaizen event. This assumes the time to measure each part is fairly short (less than 15 minutes).

In Part 2 of this blog, we will cover how to analyze the Gage R&R results and how to fix it if the measurement system fails.

How to select parts for the study.

Using Minitab statistical software to create a random sequence of parts for conducting the study.