[18806] in s-news-athena
[S] (late) summary : nls() singular gradient matrix
daemon@ATHENA.MIT.EDU (ROS Mathieu)
Mon Aug 2 04:18:37 1999
Date: Mon, 2 Aug 1999 10:08:34 +0200 (MET)
From: ROS Mathieu <mim78@cict.fr>
To: s-news@wubios.wustl.edu
Message-Id: <Pine.SGI.3.95.990802094349.7260A-100000@ondine.cict.fr>
Mime-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Sorry for summarizing so late, I was enjoying holidays :-)
Many thanks to Douglas Bates, Bill Venables, Bruce McCullough, Pat Burns
and Brian D Ripley, some always answer fast and well (hum, flowers
for you 'messieurs').
The question was originally from E. Conrad Lamon III who wrotes :
"While trying to fit a nonlinear model using nls( ), I get an error
message that says I have a singular gradient matrix."
I then ask (not very well) what was the inner reason of this error
message ("singular gradient matrix" say).
##D. Bates answered first :
"The reason that the message appears is because the gradient matrix has
become computationally singular and it is not possible to calculate the
increment for the next iteration.
The easiest way to determine what is happening within the nls function
is to add the argument
trace = TRUE
to the nls function call. With that argument you get a line printed
out at each iteration giving the current residual sum-of-squares, the
convergence criterion, and the parameter values. Most of the time the
message about singular gradients indicates that one or more of the
parameters are heading off to peculiar values, such as infinity.
One way to help convergence in nls problems is to eliminate
conditionally linear parameters with the
algorithm = "plinear"
argument. When using this you must re-write the model function as
described in the documentation.
Choosing a good parameterization and reasonable starting estimates is
also very helpful."
##Bill 'learner' Venables recalls :
"The gradient matrix, X say, is the n x p matrix of partial derivatives of
the model function with respect to the parameters, where the rows
correspond
to the observations and the the columns to the parameters. In linear
regression it is the ordinary model matrix and is constant with respect to
the parameters. In non-linear regression calculations it depends on the
current approximation to the parameter estimates and hence changes as the
iterations proceed. Locally it plays a very similar role to the model
matrix in linear regression, in fact. "Singular gradient matrix" really
means "the gradient matrix has less than full column rank" (it's a
rectangular matrix, so 'singular' or 'non-singular' are not really terms
that apply). In turn this means that the algorithm has got itself to a
point in the parameter space where there appears to be a rank loss in the
model surface, and so the parameters are (computationally at least)
locally not identified and their least squares estimates not uniquely
defined locally. In other words the iterative process does not have a
unique direction in which to look and rather than choose any one
arbitrarily it gives up.
In brief the initial values you gave it has led the algorithm to a point
on the model surface where the coordinate system provided by the
parameters appears to break down and it has no clear idea where to go
next. The only solution you have available is to start from somewhere
else and hope for better luck next time."
##B. McCullough (>>), P. Burns (>) and B.D. Ripley discuss about how to
avoid this problem :
"
>>I have found that supplying analytic first derivatives
>>using deriv() can eliminate this problem (not always,
>>but sometimes). Supplying analytic derivatives
>>is well-described in V&R.
> The optimization can be done by a more robust method such as a genetic
> algorithm. This is "robust" in the computational sense, not the
statistical
> sense. A genetic algorithm can handle non-differentiability, and
multiple
> local minima. There is a crude implementation of a genetic algorithm in
> S that is given in S Poetry.
Um. Yes, of course one could use a better optimizer, but I doubt if a
`genetic algorithm' comes into that category. Of course there are so many
`genetic algorithms' that the possibilities are unbounded, but in hard
non-linear least-squares problems (such as neural networks) they have been
tried very seriously, and seem not to be competitive. As another example,
there is a `genetic algorithm' used in the S-PLUS code for ltsreg (by one
P. Burns, I believe) and that seems to do nowhere near as well as the
standard (in that field) random search algorithms in my lqs function (in
the V&R3 library set). (`nowhere near as well' means many times slower
to achieve a larger value in a minimization problem.)
Beware of `can handle ... multiple local minima'. In what sense? There are
no known computationally feasible global optimizers for many routine
non-linear least-squares problems.
> In this setting, the use of the genetic
> algorithm would be to give a very good starting value to a more
> traditional optimizer.
An informed random multi-start approach may be as good at that.
> However, it is my experience that it is often the case that severe
> problems
> with convergence means that the data don't fit the model very well. It
> may be worth having a think about the suitability of the model from time
> to time when convergence is a problem.
Alternatively, the model may fit well only for a very narrow range of
parameters (e.g. neural networks), and the problem is to locate that
range.
Yet another potential problem is rounding due to single-precision
calculations.
It is rather hard to beat conventional optimization algorithms that make
use of first derivatives. It is quite easy to beat Gauss-Newton methods
(as used by nls) as terms they drop from the Hessian can be very
significant."
-------------------------------------------------------
Mathieu Ros
mathieuros@bigfoot.com
(Universite Paul Sabatier, Toulouse.)
Universite Joseph Fourier, Grenoble
-------------------------------------------------------
-----------------------------------------------------------------------
This message was distributed by s-news@wubios.wustl.edu. To unsubscribe
send e-mail to s-news-request@wubios.wustl.edu with the BODY of the
message: unsubscribe s-news
