[19108] in s-news-athena
[S] Summary: Confused with step.glm
daemon@ATHENA.MIT.EDU (=?iso-8859-1?Q?"G=E9rald_Jean"?=)
Fri Sep 3 14:21:58 1999
From: =?iso-8859-1?Q?"G=E9rald_Jean"?= <Gerald.Jean@spgdag.ca>
To: s-news@wubios.wustl.edu
Message-Id: <852567E1.0061C903.00@mail.spgdag.com>
Date: Fri, 3 Sep 1999 14:18:23 -0400
Mime-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
Hello S-users,
I want to acknowledge and thank, Brian Ripley, Frank E Harrell Jr, Jim
Stapleton, Andrew Robinson and Rolf Turner for shedding some light (and
supply a lot food for thoughts) on "stepwise regression". It will take me
while to digest all this and honestly the answers threw me off my shoes. I
had been using stepwise regression for quite some time and never came
across such a behavior before, but apparently it does happen and is not
uncommon.
Thanks again to all who replied, the original message along with the
replies follow.
Gérald Jean
Analyste-conseil (statistiques), Actuariat
télephone : (418) 835-8839
télecopieur : (418) 835-5865
courrier électronique: gerald.jean@spgdag.ca
"In God we trust all others must bring data"
Original message:
Dear S-users,
I have built glm's for quite a few years now; mostly logistic and probit in
the past and poisson and gamma for the last six months or so. Due to a
large number of candidate predictors I pretty much always use stepwise
methods to get down to a few models that will be analyzed in more details,
either in S+ or with home grown Fortran programs. I always notice that
stepwise methods, using criterion of fit (AIC or Hannan and Quinn), yielded
models for which the variables were significant w.r. to their t-value, or
an anova test using test = 'F' or test = 'Chi'.
Here is the problem. This time I am running a WEIGHTED poisson glm, with
log link, all the predictors are factors some with only 2 or 3 levels
others with up to 10 levels. The first variable included by step.glm has 6
levels. According to the t-value for the levels of that factor none are
significant, highest t-value amongst the levels of that factor being around
-1.35, but still it's the first one picked by step.glm. Now if I run an
anova on the output of step.glm this driver is highly significant (p-value
< 0.00000001), either with test = 'F' or test = 'Chi'. If I run glm with
the variables picked by step.glm but put this driver last the anova gives
pretty much the same results.
Waht is going on? Why is that variable not significant according to t-test
but very highly significant according to all other tests????
Please someone bring light to my confused mind!!!!
Thanks,
Brian Ripley's response:
It is known that this can happen: there is a paper by Hauck & Donner
(JASA 1977). A small t ratio means EITHER insignificant OR very
significant. See V&R2 p.237 for a more detailed explanation (than this
one). That is one reason why step.glm, which relies on local
linearization, can be almost infinitely misleadingly. It is why I
originally wrote stepAIC (in library MASS), to calculate the real AIC
for glms (and other things)
Frank E Harrell Jr. response:
Dear Gerald,
The stepwise approach invalidates all later P-values, regression
coefficient
estimates, and
standard errors. See http://www-personal.umich.edu/~dronis/statfaq.htm
Jim Stapleton's response:
I suggest that you look at the correlation matrix of the betahats
corresponding to the 6 levels (probably 5 variables), determined from
inv(X'DX), where X is the design matrix, and D is the diagonal matrix of
ests of the expected frequencies. I suspect that the correlations are quite
high. Though the subspace determined by the factor explains a significant
amount of variation in the frequencies, the high correlations cause none of
the individual variables spanning the space to be significant.
The same may be true in multiple regr., when Y = beta0 + beta1 x1 + beta2 +
ep, when x1, x2 are highly correlated. Usually the indicator variables
determining a factor, even after orthogonalizing wrt to the vector of all
ones, and other factors, are not highly correlated. In this case they may
be because of the weights. For example, most of the weights may be on the
first two levels of the factor.
Andrew Robinson's response:
When you test a variable with the t-test, you're testing the null
hypothesis that the population value for the parameter is equal to zero.
In testing continuous variables, this translates to asking whether the
variable has any explanantory power for the response variable.
Now, for factors, if you have only two levels one of them is aliased to the
overall mean, so the t-test basically has the same interpretation. If the
population value for the parameter is equal to zero then the difference
between the level being tested and the implied level (aliased to the
overall mean) is also zero. However, if you have more than two levels then
the tests suddenly mean different things. The t-test is still testing
whether or not there is any difference between the level being tested and
the overall mean, whereas all the other tests are testing the effect of the
factor on the response variable as a whole.
So, in a nutshell, ignore the t-tests for stepwise fitting of factors if
any factor has more than two levels.
Rolf Turner's response:
I believe it is fairly simple in ordinary regression for a group of
variables to have t-statistics which are not ``significant'' but
whose joint F-statistic is significant.
I.e. if you fit the model
y = a_0 + a_1*x_1 + ... + a_k*x_k + a_{k+1}*x_{k+1} + ... + a_p*x_p + Error
you might well find that none of x_{k+1}, ..., x_p has a t-statistic
that was ``significant'' at the 0.05 level, but that testing the
forgoing model against the reduced model
y = a_0 + a_1*x_1 + ... + a_k*x_k + Error
gives you an F-statistic (on p-k and n-p-1 degrees of freedom)
that ***is*** significant at the 0.05 level.
I can't give you an example off the top of my head, but my vague
recollection is that it happens ``all the time''.
Note that fitting a factor with 6 levels is equivalent to fitting 5
dummy variables.
So there is no paradox; it's just in the nature of significance
testing. To my mind this is a reason to be wary of ``automatic''
(stepwise etc.) model selection methods.
-----------------------------------------------------------------------
This message was distributed by s-news@wubios.wustl.edu. To unsubscribe
send e-mail to s-news-request@wubios.wustl.edu with the BODY of the
message: unsubscribe s-news
