****************
In my earlier article (DOE Perspectives, May 1999) I responded to Ray Myers’ paper,
which had appeared in the October 1998 issue of the Journal of Quality Technology, by
pointing out that optimal-design software now allows one to find designs that are often
far superior to standard designs, such as central-composite designs. I also pointed
out some of the features of the optimal-design-generation program I-OPT that is available
for Unix workstations from a University of Michigan web site.
One careful reader of my article thought that I should not be so single-minded in the
pursuit of optimal designs for prediction because there are other objectives of
experimentation. This reader also thought I was attempting to knock down a straw man
by picking central-composite designs and then showing that an I-optimal design could do
better for prediction. The target of my assault lacked substance for this reader
because the I-optimal designs I used were generated assuming that the model was precisely
a second-degree multivariate polynomial (a “quadratic model”) over an exactly
specified region (a multivariate cube). The reader concluded his remarks by stating
that classical designs were “constructed specifically to deal with inadequately
specified models,” and he surmised that an I-optimal design would fare poorly under
different criteria.
I consider the reader’s comments an expected sort of response, part of a lively debate
on DOE issues that will continue as computer-generated application-specific optimal
designs replace traditional practices.
I answer my reader by stating that I do not suffer from an idée fixe on optimal designs
for prediction, notwithstanding the facts that my group at the University of Michigan
developed I-OPT and our primary focus over the last decade has been on optimal designs for
prediction. Rather, my view is that technology is now making it possible for the
user to define the objectives of experimentation and to find a suitable design via
optimization. The objective may be parameter estimation, prediction, a hybrid of the
two, or something else. The method may be sequential and/or adaptive. The
missing elements are a means of obtaining the user’s objectives and software for setting
up the objective function. It is inevitable that these elements will be created in
the coming decade.
Concerning my reader’s concern that I-optimal designs are based on the assumption that
one has specified a perfectly known model function, I can point out that it is perfectly
within technological feasibility to include in the user’s input specification to an
optimal-design engine that the design be robust against model misspecification. Of
course, what is meant by “model misspecification” must be defined, but this is an
advantage because it provides the user with the opportunity to provide input on his/her
needs in this regard.
I-OPT includes one means of specifying robustness against model misspecification.
For any problem, I-OPT allows up to ten different model functions to be entered by the
user, along with the user’s best estimate of the probability that each is the correct
model. For example, a user can enter that he/she has 90% confidence that the correct
model is a full second-degree model and a 10% chance that the model is a full third-degree
model. The I-OPT search engine then uses a simple 90/10 admixture of the objective
functions for the two models considered separately in its search for the optimal design.
Gone is the requirement that the user rely on authorities telling him/her that D-optimal
or I-optimal, or whatever, are the best method. The power is given to the user.
Former authorities may be discomfited.
Regarding the reader’s statement that classical designs were “constructed specifically
to deal with inadequately specified models,” I can reply that there is no universally
good design. If there were, there would be no debate. So what is one to do to
resolve these different points of view? One answer lies in careful study of the
properties of various designs. First, there must be software available for
interested parties to find designs that meet various criteria, and then various designs
must be compared on an objective basis. The problem is that this demands a lot of
work. In this article, I point out some of the results we are obtaining at the
University of Michigan. The complete story will be published in the peer-reviewed
literature.
This past summer, I had the pleasant experience of having two outstanding undergraduate
students working with me on summer fellowships supported by my department (Electrical
Engineering and Computer Science at the University of Michigan). One was an entering
first-year student named Kimberly Kuether. Kimberly took on the challenge of finding
closed-form algebraic expressions for classical designs of experiments for second-degree
models, and she did a wonderful job of it. She learned about inverting partitioned
matrices and then applied a symbolic-manipulation program to find expressions for the
integrated variance (IV), D-optimality criterion, and A-optimality criterion for the
following types of designs: central composite designs based on both factorial and
fractional factorials, Box-Behnken designs, three-level designs in three factors, and
fractional factorials of three-level designs. The purpose here is to let people know
that such comparisons are being made and whom to contact about them. For this
article I will focus on the results from her study of central-composite designs.
After scaling the ranges of the factors so that the region of interest lies between –1
and 1 in each factor, that is, over a [-1,+1]^k hypercube, a central-composite design in k
factors consists of 2^k (2 raised to the power k) vertex points, 2k axial points a
distance “a” from the origin, and “nc” center points taken at the origin of the
region. Kimberly exploited the high degree of symmetry of these designs to find her
closed-form expressions.
One expression that she found gave the expected normalized integrated variance (IV) of the
CCD’s as a function of k, a, and nc, based on the assumption that the model is a true
second-degree function. (The expected normalized integrated variance is a measure of
how well the design will perform for predicting the response at untried locations, after
data is taken at the design points and fit with least-squares fitting. It is the
I-optimality objective.) Kimberly’s expression is complex, completely filling a
computer screen with algebra, but this could then be used to explore various behaviors of
the IV.
We could see in the algebra the well-known singularity in the IV of CCD’s for a=as=sqrt(k)
(the symbol “as” is introduced for the value of “a” where the singularity occurs),
when no center point is taken (nc=0), and we were able to determine that the singularity
behaved, for all k, as (a-sqrt(k))^(-2). This is a very pronounced singularity,
which when plotted as IV vs. “a” has a width of order unity. Put simply, the IV
of a CCD with no center point is very high, from approximately 1<a<2 for k=2 and
from 2<a<3 for k=6. This means that it is critical that there be a good datum
at the origin, as is common practice.
What is less well known, perhaps virtually unknown, is that even when one takes one or a
few center points, there is a remnant of the singularity in the IV, and this can be a
problem for users who use CCD’s for prediction. This means that the predictive
capability of a CCD will be relatively poor in the vicinity of a=sqrt(k) for all k.
We have observed that the effect persists over a width of order unity in a, just as for
the original singularity.
Consider a commonly recommended design, the rotatable CCD (the contours of expected variance of prediction for a rotatable design are hyperspheres), where the value of the distance of the star points from the origin is ar=2^[(k-p)/4]. (For completeness, I have included the integer variable “p” to indicate the fractional nature of the design. For a full-factorial CCD, p=0; for a half-fraction CCD, p=1; for a quarter-fraction CCD, p=2; etc.) A small table is given immediately below that gives the location in “a” of the singularity when nc=0, that is, at as=sqrt(k), as well as the value of “a” for the rotatable CCD, “ar.”
K |
as=sqrt(k) |
ar=2^[(k-p)/4] |
|
2 |
1.414 |
1.414 |
|
3 |
1.732 |
1.682 |
|
4 |
2.000 |
2.000 |
|
5 |
2.236 |
2.378 |
|
6 |
2.449 |
2.828 |
It can be seen that “ar” and “as” are identical for both k=2 and 4, meaning that
the rotatable CCD will be in the region of the remnant singularity for these values of k.
For other values of k of importance to industrial DOE the rotatable CCD is still in the
region of the remnant singularity. This is a warning sign.
Immediately below is a table of the expected normalized integrated variances for a few
CCD’s, with number of factors, k, ranging from 2 to 6 and the number of center points,
nc, ranging from 1 to 5. ( Readers with an avid interest in numbers may be enjoy the
observation that some of the IV values are rational numbers, as indicated in parentheses
in the table. The value of “p” is unity for all entries in this table.
K |
nc |
N |
IV at sqrt(k) |
IV at ar=2^[(k-p)/4] |
|
2 |
1 |
9 |
0.6306 (227/360) |
0.6306 (227/360) |
|
3 |
2 |
16 |
0.3701 |
0.3676 |
|
4 |
3 |
27 |
0.2667 (4/15) |
0.2667 (4/15) |
|
5 |
4 |
46 |
0.2001 |
0.1971 |
|
6 |
5 |
81 |
0.1520 |
0.1369 |
|
6 |
1 |
77 |
0.5194 |
0.2988 |
You be the judge. And remember, this is only part of the story. More study is
needed, but the results that my group has found are instructive. One can use an
optimal-design engine, such as I-OPT or gosset to find the design that minimizes the IV
over the region of interest, i.e., [-1,+1]^k, allowing that the design points can expand
out to the same value of “a” that is used in the CCD.
For k=2, the N=9, I-optimal design is the following (or any of three equivalently good
designs found by rotating this design by 90, 180, or 270 degrees about the origin):
[4x(-0.098, +0.098), (-1.000, -0.828), (-1.000, +1.000), (+1.000, -0.336), (+0.828,
+1.000), (+0.336, -1.000)]. The integrated variance of this design is 0.2945, which
is more than a factor of two lower that the IV for the rotatable CCD (0.6306).
In more factors, the benefit of using an I-optimal design as opposed to a rotatable,
full-factorial CCD grows rapidly. By k=6, the factor of improvement is greater than
a factor of ten. This means that you have the choice of using an I-optimal design
with the same number of points as the corresponding rotatable CCD and you can expect
to get a better fit, or you may choose to use an I-optimal design with many fewer points,
since the minimum number of points for a full second-degree I-optimal design in six
factors is only 28. So you could use, say 32, instead of the 77 to 81 points for the
CCD, and still expect to have a better prediction, since the IV’s of the optimal designs
scale roughly linearly with N.
Let me add that if the objective of experimentation is not prediction over the entire
region of the factor space, then another objective may be more appropriate, such as
D-optimality. Kimberly has found that the D-optimal designs also perform much better
than CCD’s and other classical designs, when the objective is minimizing the error in
the parameter estimates. Thus, what we are observing seems generic to optimal
designs. With a properly defined objective function optimal designs generally
perform much better than classical designs.
The bottom line is that optimal designs can improve your experimentation greatly and/or
reduce the amount of experimentation needed to succeed in your objective.
Since I’m on vacation in Denmark right now, I don’t have all the numbers from all of
my group’s work available, but the comprehensive results will be included in a
forthcoming report. Anyone wishing a few numbers from my group may contact me.
****************
Note that our limited resources mean that these are the ONLY public classes that we are
offering through the end of the year. Space is limited so sign up now. Note
that completion of an approved basic class is a pre-requisite for attending the advanced
class.
These classes are sponsored and run by Math Options (http://www.mathoptions.com).
Check the Math Options web site for full details.
Las Vegas, NV
Nov. 1
(AKA A Modern DOE Workshop)
Las Vegas, NV
Nov. 2-4
****************
The free teleclass “How to Tell the Truth with Statistics” with Bill Kappele will be
offered by Math Options on October 26, 1999, at 10 AM Pacific time (1:00 Eastern time).
For more information and to register please call
Bill Kappele at (888) 764-3958 or visit
--
Bill Kappele also passes along this tip: Note that reasonable is high praise coming
from Bill.
Thomas Pyzdek’s article in the August issue of “Quality Digest” is a an excellent
article. Mr. Pyzdek discusses the dilemma faced by all researchers, namely, “How
can I utilize data that wasn’t collected using DOE?” His conclusions are
extremely reasonable.
You can find this article, “Applying Virtual DOE in the Real World,” at http://www.qualitydigest.com/aug99/html/sixsig.html
****************
Several designs have been added to the Math Options I-Optimal Design Library. These
are:
1. Process Factor designs for spherical regions.
2. Designs for mixtures and discrete factors.
I am also in the process of adding information about the designs, specifically the log
condition number, the maximum VIF, the model, and whether the region is cubical or
spherical. You will find this information for some of the designs now.
Access to the library will now be much easier! Those of you who download the library
periodically will no longer need to provide contact information every time! At the
top right corner of the page for the Library, http://www.mathoptions.com/i-optima1.htm,
you will find an entrance specifically for you in red. This should make your life
easier!
You can also download the entire library with one click now. You will need a
password to unzip the file. The password is mailed to first-time registrants when
they
****************
An R&D unit based just outside of Paris is looking for someone to head up their
Molecular Modeling team. For complete details, contact Marie De Maistre at mailto:transearch@transearch.com.
****************
XIV Compstat
Utrecht, The Netherlands
21-25 August 2000
http://neon.vb.cbs.nl/rsm/compstat
* * * * * * *
*
Alan Brice Corwin
7349 16th Avenue NE
Seattle, WA 98115
206-525-9093
mailto:abcorwin@processbuilder.com
http://www.processbuilder.com/