Research Article

ESD Reports Summer 2005

Experimentation and Engineering Systems

Helping to make the ESD Doughnut Whole

By Daniel D. Frey, Robert N. Noyce Career Development Professor, Assistant Professor of Mechanical Engineering and Engineering Systems

A common topic of discussion among ESD faculty and researchers is the “ESD doughnut hole” – establishing the core of the theory that we hope will advance engineering systems as a discipline.

I propose that one promising way to fill that doughnut hole is to identify existing foundations in engineering and science where there is potential for revolution in the sense described by Thomas Kuhn in The Structure of Scientific Revolutions. According to Kuhn’s account of the history of science, there is an activity called “normal science” in which researchers build upon existing foundations. However, there are also scientific revolutions in which existing foundations are challenged and replaced by substantially new foundations.

Three signs are typically observed during the onset of scientific revolutions:

1. Vigorous disputes resulting in little convergence. For example, at the onset of the Copernican revolution in astronomy, it was not possible to resolve some disagreements by rational discourse. Some scientists simply could not accept that the earth moved.
2. Unresolved anomalies. For example, before Einstein’s theories of relativity were proposed, the concept of the ether was clashing with apparent constancy in measurements of the speed of light in different directions.
3. Persistent failures in transition to practice. Behaviorism was the dominant model in psychology for much of the early 20th century, but its failure to produce much useful insight into the practice of clinical psychology was an important contributor to its downfall.

I am convinced that all of these leading indicators have been observed when Design of Experiments (DOE) is applied to engineering systems. DOE is a technical discipline for planning experiments, analyzing resulting data, and drawing conclusions from that analysis.

DOE traces it roots to R. A. Fisher who, motivated by the challenges of efficient agricultural experimentation, first proposed factorial experimentation and analysis of variance. Fischer’s methods were extremely helpful to those seeking, for example, to find new combinations of nutrients, pesticides, and watering conditions that would provide more productive harvests. Subsequently, a community of statistical researchers made DOE increasingly sophisticated and useful.

Despite impressive progress, a crisis may be emerging concerning DOE. The three signs that revolution seems to be in evidence are:

1. Vigorous disputes resulting in little convergence. After WWII, Genichi Taguchi pioneered methods to make systems more robust and developed them further through working closely with industry. The statistics community found many of Taguchi’s methods problematic and much debate ensued, but neither side substantially altered their position.
2. Unresolved anomalies. Statistics researchers sought to improve upon Taguchi’s methods and offered very convincing arguments to support a different approach. Yet some of these theoretically superior approaches don’t seem to work better in practice. For example, recent publications by multiple independent investigators show that Taguchi’s crossed arrays work better than theoretically preferable combined arrays.
3. Persistent failures in transition to practice. A renowned statistician, George E. P. Box, has argued that mathematical formality in DOE has led to overuse of “optimal” experimental designs that discourage iteration. In other words, Box has said that even though theory is suggesting you should run larger experiments, experience suggests you should run multiple iterations of smaller experiments.

Based on these observations, I feel that DOE may soon experience a scientific revolution and that its foundations will be substantially modified as a result. This hunch has motivated a reexamination of DOE’s foundations.

The table below shows example of the results. The columns correspond to two types of models, one found in DOE textbooks, another developed based on 113 data sets. The table rows correspond to alternative robust design methods. The values in the table indicate the degree of robustness improvement typically provided by the method under the given scenario.

In this research, Taguchi’s crossed arrays have been shown to be superior to combined arrays, which is consistent with what head-to-head comparisons have shown. In addition, an even more effective method emerged that uses more iteration and therefore roughly doubles the benefits accomplished through Taguchi methods.

To summarize, although DOE provides an existing basis of theory for engineering systems, it also exhibits many signs of crisis. In seeking to resolve aspects of this crisis, our research group has worked to broaden the theoretical basis of DOE. Therefore, in a small way, our research may help to address the ESD doughnut hole.

How much improvement do robust design methods provide? It depends what you assume about the system you apply them to. A new model based on ESD research seems to resolve an old argument between theorists and practitioners. The insights from the research also led to a new method much better than either previously existing alternative.