When we’re looking at results (like measures of a characteristic), we need to take care not to get too hung-up on what the statistics is trying to tell us. Yes, statistical tools are a good way for us to make decisions and the results can act as proof for us. But, there’s a practical, engineering side to results, too. We need to evaluate the statistical significance along with the practical significance.
We review an example and how to document it.
We use tools to help us make decisions and predict outcomes with some confidence. Some of the techniques are a way for us to apply a mathematical model, like statistics. Some are graphical organizers. And others, still, help a team of people organize and coordinate their ideas. They’re all tools for people to use to make decisions. Ultimately, we use the results of many inputs in order to make our best decisions.
It is important to distinguish between statistical significance and practical significance. Statistical significance simply means that we reject the null hypothesis. The ability of the test to detect differences that lead to rejection of the null hypothesis depends on the sample size. For example, for a particularly large sample, the test may reject the null hypothesis that two process means are equivalent. However, in practice the difference between the two means may be relatively small to the point of having no real engineering significance. Similarly, if the sample size is small, a difference that is large in engineering terms may not lead to rejection of the null hypothesis. The analyst should not just blindly apply the tests, but should combine engineering judgement with statistical analysis. (NIST)
Citations & References:
"1.3.5 Quantitative Techniques, Practical Versus Statistical Significance". NIST/SEMATECH e-Handbook of Statistical Methods. NIST. www.itl.nist.gov/div898/handbook//eda/section3/eda35.htm. Accessed Oct. 16, 2021.
Montgomery, Douglas C. Design and Analysis of Experiments, 5th Ed. John Wiley & Sons, Inc. 2001, pp 19.