[6377] in s-news-athena
choosing smoother df for smooth.spline
daemon@ATHENA.MIT.EDU (Bill Shipley)
Fri Feb 3 10:59:09 1995
Date: Fri, 3 Feb 1995 10:01:09 -0500
To: S-news@utstat.toronto.edu
From: bshipley@courrier.usherb.ca (Bill Shipley)
This is an undate on the posting I did a few days ago concerning confidence
intervals of the 1st derivatives obtained by cubic spline smoothers
(smooth.spline(cv=T) followed by predict.smooth.spline(deriv=1)). I noted
that the use of cross-validation (i.e. cv=T) to choose the smoother degrees
of freedom can occasionally give very wrong answers (Hastie and Tibshirani's
book shows the same thing). However, I have found that a good way of
improving the behaviour of the cross-validation df is to bootstrap the data
and use the modal value of the bootstraped smother degrees of freedom.
For instance, I have been exploring the ability of smooth.spline() to
capture a function whose 1st derivatives follow a beta function. When I
generate data that follow this function but with normally distributed
variation, I can occasionally find sample data sets that wildly undersmooth
(df=16 when it should be around 7 or 8). However, when I do 500 bootstrap
samples from this wild data set and plot a histogram of the 500 df, the mode
is 8.
I have no theoretical justification for this proceedure, but it does seem to
work well. Now, if I can only find a way to remove the bias in the
estimated derivatives near the extremes of the data....
Any comments?
Bill Shipley
Departement de Biologie
Universite de Sherbrooke
Sherbrooke (Quebec)
CANADA J1K 2R1
bshipley@courrier.USherb.ca
