[19163] in s-news-athena
Re: [S] Comparing poisson, quasi-log-mu, and negative binomial glm
daemon@ATHENA.MIT.EDU (Prof Brian D Ripley)
Thu Sep 9 03:11:18 1999
Date: Thu, 9 Sep 1999 08:03:31 +0100 (BST)
From: Prof Brian D Ripley <ripley@stats.ox.ac.uk>
To: Bill Venables <William.Venables@cmis.CSIRO.AU>
Cc: s-news@wubios.wustl.edu
In-Reply-To: <199909090015.KAA06194@snowy.nsw.cmis.CSIRO.AU>
Message-Id: <Pine.GSO.4.05.9909090745440.3933-100000@auk.stats>
Mime-Version: 1.0
Content-Type: TEXT/PLAIN; charset=X-UNKNOWN
Content-Transfer-Encoding: 8bit
There are a couple of special points here. First, as Poisson and negative
binomial distributions are for discrete data, there is a very natural
normalization for the log-likelihood: just add up the log probabliities of
the observed counts. This does make them comparable, and allows penalized
methods such as AIC and SBC to be used. Indeed, in this context, I think
they have superseded the Cox `separate families' approach (and did in a
talk David Cox gave a couple of years ago on model selection).
[I agree that the normalization of log-likelihoods is usually arbitrary,
but for discrete data they are equivalent in the most obvious normalization
to log-probability scoring, and that is not arbitrary.]
Second, a `quasi' glm is not a complete probability model so it cannot make
predictions of probabilities, only estimate the variation of means with
covariates. It is thus non-comparable unless you are only interested in
means and variances. If you are you can set up a test using only those.
if you are interested in the probability of a zero count, for example,
you need a real model.
Generally, on model comparison. There is a large literature out there,
and I would summarize it by saying you need to test that models are fit for
your purpose. Often a good way to do this will be to do a cross-validation
test of the predictions of something you are actually interested in.
On Thu, 9 Sep 1999, Bill Venables wrote:
[Quoting someone his mailer has not reproduced]
> >
> > Dear Members of the S+ community:
> >
> > For the analysis of count data, the Splus glm function offers, at
> > least, three useful alternatives, analyzable in three main families:
> > Poisson, negative binomial, and quasi (with a log-link & a
> > mu-variance). The negative binomial option is available in the glm.nb
> > function of the Venables & Ripley mass library. The glm.nb and the
> > quasi-log-mu are useful for overdispersed data.
>
> ....
>
> >
> > Questions:
> > 1. How to compare, particularly the negative binomial and quasi-log-mu
> > models? What criteria are most advisable to indicate that one Splus glm
> > model is superior to the other?
>
> A few picky points first. It's not an `Splus glm model'. Splus is a piece of
> software with nothing to do with models and `glm' already has the word "model"
> embedded in it, so `glm model' is in the same category as `PIN number'....
> Also negative binomial models are, strictly speaking, outside the glm family,
> unless the theta parameter is known.
>
> Now for more serious issues. For the same linear structure negative binomial
> and quasi-likelihood models with log link and variance proportional to the mean
> have the same numbers of parameters, and comparing models with the same
> parametric degree can be subtle. It requires you to say very clearly what you
> really mean by one model being "superior" to the other. Presented with this
> problem I think my first approach would be informal and fairly graphical. How
> well both models shape up for prediction would be a primary consideration and
> you can clearly explore that issue by all kinds of informal computational and
> graphical methods.
>
> There is an old paper by David Cox on this issue, known as "testing separate
> families of hypotheses" that has periodically generated some sparks of interest
> but the idea has really yet to take off very seriously. A few references are
> given below (but not the original of David Cox).
>
> > 2. I saw, for example, that some authors (who use other software)
> > evaluate the models through a comparison of the models’
> > log-likelihoods. Are such comparisons appropriate?
>
> Likelihood ratio tests are primarily for nested hypotheses, that is, when the
> inner model is strictly a special case of the outer. Since likelihood is only
> defined up to a multiplicative constant, using likelihood to compare non-nested
> models (as you have here) first requires you to decide on some way of
> normalizing them so that they can be compared. This could be done by defining
> a super model that contains both of your models as special cases. With one
> model a genuine parametric model and the other as a quasi-model, this dodge is
> likely to be an interesting if minor research question. Over to you.
>
> > 3. If comparisons of log-likelihoods would be appropriate, the Splus
> > “summary” does not provide the log-likelihoods for the Poisson and the
> > quasi-log-mu models. Other software packages (e.g., LIMDEP) routinely
> > provide log-likelihoods for the Poisson models. Can log-likelihoods be
> > generated for the Splus Poisson and quasi-log-mu models? If so, how?
>
> Routinely providing log-likelihoods is useless unless you know how they have
> been normalized.
>
> For Poisson models the deviance *is* a log-likelihood, normalized so that it is
minus twice log-likelihood, I think.
> zero at the saturated model. This normalization is not possible for negative
> binomial models, though, since the parameters are not identified at the
> saturated model (in fact you could choose to estimate theta by equating the
> deviance to n-p, as some authors do; the deviance provided by glm.nb is
> generally of no interest, except as a very informal and rough test of fit by
> comparing it with n-p). For quasi-likelihood models the deviance is *not* a
> log-likelihood, not even a "log-quasi-likelihood", (though it would be if the
> scale parameter were known and its value 1, as is the case for Poisson).
>
> Of course it is (trivially) possible to provide log-likelihoods (or
> log-quasi-likelihoods) for all of these models, but before you do so you must
> first come to some agreement, again, on how they should be normalized and that
> really is quite arbitrary.
[Not for count data, I think. Deviances were put there to confuse us....]
[ refs removed ]
--
Brian D. Ripley, ripley@stats.ox.ac.uk
Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/
University of Oxford, Tel: +44 1865 272861 (self)
1 South Parks Road, +44 1865 272860 (secr)
Oxford OX1 3TG, UK Fax: +44 1865 272595
-----------------------------------------------------------------------
This message was distributed by s-news@wubios.wustl.edu. To unsubscribe
send e-mail to s-news-request@wubios.wustl.edu with the BODY of the
message: unsubscribe s-news
